"2020-0831|CEF, E20 ITMAL update, fixed typo in y.shape and make gfx links to BB.\n",

"2021-0201|CEF, F21 ITMAL update.\n",

"\n",

"### Mini Python/Jupyternotebook demo\n",

"\n",

"Build-in python array an Numpy arrays..."

...

...

@@ -83,9 +74,10 @@

"\n",

"Just use Numpy as a matrix like class; create a (3 x 4) matrix and do some matrix operations on it... \n",

"\n",

"<img src='https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L01/Figs/matrix.jpg' alt=\"WARNING: you need to be logged into Blackboard to view images\">\n",

"<img src='https://itundervisning.ase.au.dk/GITMAL/L01/Figs/matrix.jpg' alt=\"WARNING: you need to be logged into Blackboard to view images\">\n",

"\n",

"(NOTE: do not use `numpy.matrix`, <a href='https://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html'>it is unfortunatly depricated.</a>)"

"(NOTE: do not use `numpy.matrix`, <a href='https:\n",

" //docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html'>it is unfortunatly depricated.</a>)"

]

},

{

...

...

@@ -191,6 +183,23 @@

"\n",

"print(\"\\nOK\")"

]

},

{

"cell_type": "markdown",

"metadata": {},

"source": [

"## Administration\n",

"\n",

"REVISIONS||\n",

"---------||\n",

"2019-01-28| CEF, initial. \n",

"2019-08-20| CEF, E19 ITMAL update. \n",

"2019-08-28| CEF, split into more cells.\n",

"2020-01-25| CEF, F20 ITMAL update.\n",

"2020-08-31| CEF, E20 ITMAL update, fixed typo in y.shape and make gfx links to BB.\n",

"2021-02-01| CEF, F21 ITMAL update.\n",

"2021-08-02| CEF, update to E21 ITMAL.\n"

]

}

],

"metadata": {

...

...

@@ -233,9 +242,9 @@

},

"position": {

"height": "298.85px",

"left": "1134px",

"left": "766px",

"right": "20px",

"top": "95px",

"top": "218px",

"width": "452.333px"

},

"types_to_exclude": [

...

...

@@ -245,7 +254,7 @@

"instance",

"_Feature"

],

"window_display": true

"window_display": false

}

},

"nbformat": 4,

...

...

%% Cell type:markdown id: tags:

# ITMAL Intro

## Mini Python Demo

REVISIONS||

---------||

2019-0128|CEF, initial.

2019-0820|CEF, E19 ITMAL update.

2019-0828|CEF, split into more cells.

2020-0125|CEF, F20 ITMAL update.

2020-0831|CEF, E20 ITMAL update, fixed typo in y.shape and make gfx links to BB.

2021-0201|CEF, F21 ITMAL update.

### Mini Python/Jupyternotebook demo

Build-in python array an Numpy arrays...

%% Cell type:code id: tags:

``` python

%reset-f

# import clause, imports numpy as the name 'np'

importnumpyasnp

# python build-in array

x=[[1,2,3],[4,5,6]]

# print using print-f-syntax, prefeed againts say print('x = ',x)

print(y[0,-1])# elem 0-from the 'right', strange but pythonic

print(y[0,-2])# elem 1-from the 'right'

# print a column, but will display as 'row'

print(y[:,1])

print('\nOK')

```

%% Cell type:markdown id: tags:

#### Matrix multiplication

Just use Numpy as a matrix like class; create a (3 x 4) matrix and do some matrix operations on it...

<imgsrc='https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L01/Figs/matrix.jpg'alt="WARNING: you need to be logged into Blackboard to view images">

<imgsrc='https://itundervisning.ase.au.dk/GITMAL/L01/Figs/matrix.jpg'alt="WARNING: you need to be logged into Blackboard to view images">

(NOTE: do not use `numpy.matrix`, <ahref='https://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html'>it is unfortunatly depricated.</a>)

(NOTE: do not use `numpy.matrix`, <a href='https:

//docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html'>it is unfortunatly depricated.</a>)

"The train-predict (or train-fit) process on some data can be visualized as\n",

"\n",

"<img src=\"https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L01/Figs/supervised_learning.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"height:250px\">\n",

"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L01/Figs/supervised_learning.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"height:250px\">\n",

"\n",

"In this figure the untrained model is a `sklearn.linear_model.LinearRegression` python object. When trained via `model.fit()`, using some know answers for the data, $\\mathbf{y}_{true}~$, it becomes a blue-boxed trained model.\n",

"\n",

...

...

@@ -452,7 +452,7 @@

"\n",

"Hint here is a similar plot of a KNN for a small set of different k's:\n",

"\n",

"<img src=\"https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-91831/Fildeling/L01/Figs/regression_with_knn.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"height:150px\">"

"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L01/Figs/regression_with_knn.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"height:150px\">"

]

},

{

...

...

@@ -550,18 +550,19 @@

"source": [

"REVISIONS||\n",

"---------||\n",

"2018-1218|CEF, initial. \n",

"2019-0124|CEF, spell checked and update. \n",

"2019-0130|CEF, removed reset -f, did not work on all PC's. \n",

"2019-0820|CEF, E19 ITMAL update. \n",

"2019-0826|CEF, minor mod to NN exercise.\n",

"2019-0828|CEF, fixed dataset dir issue, datapath\"../datasets\" changed to \"./datasets\".\n",

"2020-0125|CEF, F20 ITMAL update.\n",

"2020-0806|CEF, E20 ITMAL update, minor fix of ls to dir and added exception to datasets load, udpated figs paths.\n",

"2020-0924|CEF, updated text to R2, Qa exe.\n",

"2020-0928|CEF, updated R2 and theta extraction, use python attributes, moved revision table. Added comment about MLP.\n",

"2021-0112|CEF, updated Qe.\n",

"2011-0208|CEF, added ls for Mac/Linux to dir command cell. "

"2018-12-18|CEF, initial. \n",

"2019-01-24|CEF, spell checked and update. \n",

"2019-01-30|CEF, removed reset -f, did not work on all PC's. \n",

"2019-08-20|CEF, E19 ITMAL update. \n",

"2019-08-26|CEF, minor mod to NN exercise.\n",

"2019-08-28|CEF, fixed dataset dir issue, datapath\"../datasets\" changed to \"./datasets\".\n",

"2020-01-25|CEF, F20 ITMAL update.\n",

"2020-08-06|CEF, E20 ITMAL update, minor fix of ls to dir and added exception to datasets load, udpated figs paths.\n",

"2020-09-24|CEF, updated text to R2, Qa exe.\n",

"2020-09-28|CEF, updated R2 and theta extraction, use python attributes, moved revision table. Added comment about MLP.\n",

"2021-01-12|CEF, updated Qe.\n",

"2021-02-08|CEF, added ls for Mac/Linux to dir command cell. \n",

"2021-08-02|CEF, update to E21 ITMAL."

]

}

],

...

...

%% Cell type:markdown id: tags:

# ITMAL Exercise

## Intro

We startup by reusing parts of `01_the_machine_learning_landscape.ipynb` from Géron [GITHOML]. So we begin with what Géron says about life satisfactions vs GDP per capita.

Halfway down this notebook, a list of questions for ITMAL is presented.

%% Cell type:markdown id: tags:

## Chapter 1 – The Machine Learning landscape

_This is the code used to generate some of the figures in chapter 1._

%% Cell type:markdown id: tags:

### Setup

First, let's make sure this notebook works well in both python 2 and 3, import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures:

This function just merges the OECD's life satisfaction data and the IMF's GDP per capita data. It's a bit too long and boring and it's not specific to Machine Learning, which is why I left it out of the book.

print(f"ITMAL NOTE: well, you need to have the 'datasets' dir in path, please unzip 'datasets.zip' and make sure that its included in the datapath='{datapath}' setting in the cell above..")

## Ultra-brief Intro to the Fit-Predict Interface in Scikit-learn

OK, the important lines in the cells above are really just

```python

#Select a linear model

model=sklearn.linear_model.LinearRegression()

# Train the model

model.fit(X,y)

# Make a prediction for Cyprus

X_new=[[22587]]# Cyprus' GDP per capita

y_pred=model.predict(X_new)

print(y_pred)# outputs [[ 5.96242338]]

```

What happens here is that we create model, called LinearRegression (for now just a 100% black-box method), put in our data training $\mathbf{X}$ matrix and corresponding desired training ground thruth vector $\mathbf{y}$ (aka $\mathbf{y}_{true})$, and then train the model.

After training we extract a _predicted_ $\mathbf{y}_{pred}$ vector from the model, for some input scalar $x$=22587.

### Supervised Training via Fit-predict

The train-predict (or train-fit) process on some data can be visualized as

<imgsrc="https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L01/Figs/supervised_learning.png"alt="WARNING: you need to be logged into Blackboard to view images"style="height:250px">

<imgsrc="https://itundervisning.ase.au.dk/GITMAL/L01/Figs/supervised_learning.png"alt="WARNING: you need to be logged into Blackboard to view images"style="height:250px">

In this figure the untrained model is a `sklearn.linear_model.LinearRegression` python object. When trained via `model.fit()`, using some know answers for the data, $\mathbf{y}_{true}~$, it becomes a blue-boxed trained model.

The trained model can be used to _predict_ values from new, yet-unseen, data, via the `model.predict()` function.

In other words, how high is life-satisfaction for Cyprus' GDP=22587 USD?

Just call `model.predict()` on a matrix with one single numerical element, 22587, well, not a matrix really, but a python list-of-lists, `[[22587]]`

```y_pred = model.predict([[22587]])```

Apparently 5.96 the models answers!

(you get used to the python built-in containers and numpy on the way..)

%% Cell type:markdown id: tags:

### Qa) The $\theta$ parameters and the $R^2$ Score

Géron uses some $\theta$ parameter from this linear regression model, in his examples and plots above.

How do you extract the $\theta_0$ and $\theta_1$ coefficients in his life-satisfaction figure form the linear regression model, via the models python attributes?

Read the documentation for the linear regressor at

Now comes the really fun part: all methods in Scikit-learn have this fit-predict interface, and you can easily interchange models in your code just by instantiating a new and perhaps better ML model.

There are still a lot of per-model parameters to tune, but fortunately, the built-in default values provide you with a good initial guess for good model setup.

Later on, you might want to go into the parameter detail trying to optimize some params (opening the lid of the black-box ML algo), but for now, we pretty much stick to the default values.

Let's try to replace the linear regression now, let's test a _k-nearest neighbour algorithm_ instead (still black boxed algorithm-wise)...

### Qb) Using k-Nearest Neighbors

Change the linear regression model to a `sklearn.neighbors.KNeighborsRegressor` with k=3 (as in [HOML:p21,bottom]), and rerun the `fit` and `predict` using this new model.

What do the k-nearest neighbours estimate for Cyprus, compared to the linear regression (it should yield=5.77)?

What _score-method_ does the k-nearest model use, and is it comparable to the linear regression model?

Seek out the documentation in Scikit-learn, if the scoring methods are not equal, can they be compared to each other at all then?

Remember to put pointer/text from the Sckikit-learn documentation in the journal...(did you find the right kNN model etc.)

%% Cell type:code id: tags:

``` python

# this is our raw data set:

sample_data

```

%% Cell type:code id: tags:

``` python

# and this is our preprocessed data

country_stats

```

%% Cell type:code id: tags:

``` python

# Prepare the data

X = np.c_[country_stats["GDP per capita"]]

y = np.c_[country_stats["Life satisfaction"]]

print("X.shape=",X.shape)

print("y.shape=",y.shape)

# Visualize the data

country_stats.plot(kind='scatter', x="GDP per capita", y='Life satisfaction')

plt.show()

# Select and train a model

# TODO: add your code here..

assert False, "TODO: add you instatiation and training of the knn model here.."

# knn = ..

```

%% Cell type:markdown id: tags:

### Qc) Tuning Parameter for k-Nearest Neighbors and A Sanity Check

But that not the full story. Try plotting the prediction for both models in the same graph and tune the `k_neighbor` parameter of the `KNeighborsRegressor` model.

Choosing `k_neighbor=1` produces a nice `score=1`, that seems optimal...but is it really so good?

Plotting the two models in a 'Life Satisfaction-vs-GDP capita' 2D plot by creating an array in the range 0 to 60000 (USD) (the `M` matrix below) and then predict the corresponding y value will sheed some light to this.

Now reusing the plots stubs below, try to explain why the k-nearest neighbour with `k_neighbor=1` has such a good score.

Does a score=1 with `k_neighbor=1`also mean that this would be the prefered estimator for the job?

Hint here is a similar plot of a KNN for a small set of different k's:

<img src="https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-91831/Fildeling/L01/Figs/regression_with_knn.png" alt="WARNING: you need to be logged into Blackboard to view images" style="height:150px">

<img src="https://itundervisning.ase.au.dk/GITMAL/L01/Figs/regression_with_knn.png" alt="WARNING: you need to be logged into Blackboard to view images" style="height:150px">

%% Cell type:code id: tags:

``` python

sample_data.plot(kind='scatter', x="GDP per capita", y='Life satisfaction', figsize=(5,3))

plt.axis([0, 60000, 0, 10])

# create an test matrix M, with the same dimensionality as X, and in the range [0;60000]

# and a step size of your choice

m=np.linspace(0, 60000, 1000)

M=np.empty([m.shape[0],1])

M[:,0]=m

# from this test M data, predict the y values via the lin.reg. and k-nearest models

y_pred_lin = model.predict(M)

y_pred_knn = knn.predict(M) # ASSUMING the variable name 'knn' of your KNeighborsRegressor

# use plt.plot to plot x-y into the sample_data plot..

plt.plot(m, y_pred_lin, "r")

plt.plot(m, y_pred_knn, "b")

# TODO: add your code here..

assert False, "TODO: try knn with different k_neighbor params, that is re-instantiate knn, refit and replot.."

```

%% Cell type:markdown id: tags:

### Qd) Trying out a Neural Network

Let us then try a Neural Network on the data, using the fit-predict interface allows us to replug a new model into our existing code.

There are a number of different NN's available, let's just hook into Scikit-learns Multi-Layer Perceptron for regression, that is an 'MLPRegressor'.

Now, the data-set for training the MLP is really not well scaled, so we need to tweak a lot of parameters in the MLP just to get it to produce some sensible output: with out preprocessing and scaling of the input data, `X`, the MLP is really a bad choice of model for the job since it so easily produces garbage output.

Try training the `mlp` regression model below, predict the value for Cyprus, and find the `score` value for the training set...just as we did for the linear and KNN models.

Can the `MLPRegressor` score function be compared with the linear and KNN-scores?

%% Cell type:code id: tags:

``` python

from sklearn.neural_network import MLPRegressor

# Setup MLPRegressor, can be very tricky for the tiny-data

# lets make a MLP regressor prediction and redo the plots

y_pred_mlp = mlp.predict(M)

plt.plot(m, y_pred_lin, "r")

plt.plot(m, y_pred_knn, "b")

plt.plot(m, y_pred_mlp, "k")

# TODO: add your code here..

assert False, "TODO: predict value for Cyprus and fetch the score() from the fitting."

```

%% Cell type:markdown id: tags:

### [OPTIONAL] Qe) Neural Network with pre-scaling

Now, the neurons in neural networks normally expects input data in the range `[0;1]` or sometimes in the range `[-1;1]`, meaning that for value outside this range the you put of the neuron will saturate to it's min or max value (also typical `0` or `1`).

A concrete value of `X` is, say 22.000 USD, that is far away from what the MLP expects. To af fix to the problem in Qd) is to preprocess data by scaling it down to something more sensible.

Try to scale X to a range of `[0;1]`, re-train the MLP, re-plot and find the new score from the rescaled input. Any better?

%% Cell type:code id: tags:

``` python

# TODO: add your code here..

assert False, "TODO: try prescale data for the MPL...any better?"

```

%% Cell type:markdown id: tags:

REVISIONS||

---------||

2018-1218|CEF, initial.

2019-0124|CEF, spell checked and update.

2019-0130|CEF, removed reset -f, did not work on all PC's.

2019-0820|CEF, E19 ITMAL update.

2019-0826|CEF, minor mod to NN exercise.

2019-0828|CEF, fixed dataset dir issue, datapath"../datasets" changed to "./datasets".

2020-0125|CEF, F20 ITMAL update.

2020-0806|CEF, E20 ITMAL update, minor fix of ls to dir and added exception to datasets load, udpated figs paths.

2020-0924|CEF, updated text to R2, Qa exe.

2020-0928|CEF, updated R2 and theta extraction, use python attributes, moved revision table. Added comment about MLP.

2021-0112|CEF, updated Qe.

2011-0208|CEF, added ls for Mac/Linux to dir command cell.

2018-12-18|CEF, initial.

2019-01-24|CEF, spell checked and update.

2019-01-30|CEF, removed reset -f, did not work on all PC's.

2019-08-20|CEF, E19 ITMAL update.

2019-08-26|CEF, minor mod to NN exercise.

2019-08-28|CEF, fixed dataset dir issue, datapath"../datasets" changed to "./datasets".

2020-01-25|CEF, F20 ITMAL update.

2020-08-06|CEF, E20 ITMAL update, minor fix of ls to dir and added exception to datasets load, udpated figs paths.

2020-09-24|CEF, updated text to R2, Qa exe.

2020-09-28|CEF, updated R2 and theta extraction, use python attributes, moved revision table. Added comment about MLP.

2021-01-12|CEF, updated Qe.

2021-02-08|CEF, added ls for Mac/Linux to dir command cell.

"For ___Windows___, you have to add `PYTHONPATH` to your user environment variables...see screenshot below (enlarge by modding the image width-tag or find the original png in the Figs directory).\n",

"\n",

"<img src=\"https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L01/Figs/Screenshot_windows_enviroment_variables.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:350px\">\n",

"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L01/Figs/Screenshot_windows_enviroment_variables.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:350px\">\n",

"\n",

"or if you, like me, hate setting up things in a GUI, and prefer a console, try in a CMD on windows\n",

"\n",

...

...

@@ -283,17 +283,18 @@

"\n",

"REVISIONS||\n",

"---------||\n",

"2018-1219| CEF, initial. \n",

"2018-0206| CEF, updated and spell checked. \n",

"2018-0207| CEF, made Qh optional.\n",

"2018-0208| CEF, added PYTHONPATH for windows.\n",

"2018-0212| CEF, small mod in itmalutils/utils.\n",

"2020-09-07| CEF, added text on OPRG and OOP for EE's\n",

"2020-09-29| CEF, added elaboration for journal in Qa+b.\n",

"2021-02-06| CEF, fixed itmalutils.TestAll() in markdown cell.\n",

"2021-08-02| CEF, update to E21 ITMAL."

]

}

],

...

...

%% Cell type:markdown id: tags:

# ITMAL Exercise

## Python Basics

### Modules and Packages in Python

Reuse of code in Jupyter notebooks can be done by either including a raw python source as a magic command

```python

%loadfilename.py

```

but this just pastes the source into the notebook and creates all kinds of pains regarding code maintenance.

A better way is to use a python __module__. A module consists simply (and pythonic) of a directory with a module init file in it (possibly empty)

```python

libitmal/__init__.py

```

To this directory you can add modules in form of plain python files, say

```python

libitmal/utils.py

```

That's about it! The `libitmal` file tree should now look like

```

libitmal/

├── __init__.py

├── __pycache__

│ ├── __init__.cpython-36.pyc

│ └── utils.cpython-36.pyc

├── utils.py

```

with the cache part only being present once the module has been initialized.

You should now be able to use the `libitmal` unit via an import directive, like

```python

importnumpyasnp

fromlibitmalimportutilsasitmalutils

print(dir(itmalutils))

print(itmalutils.__file__)

X=np.array([[1,2],[3,-100]])

itmalutils.PrintMatrix(X,"mylabel=")

itmalutils.TestAll()

```

#### Qa Load and test the `libitmal` module

Try out the `libitmal` module from [GITMAL]. Load this module and run the function

```python

fromlibitmalimportutilsasitmalutils

itmalutils.TestAll()

```

from this module.

##### Implementation details

Note that there is a python module ___include___ search path, that you may have to investigate and modify. For my Linux setup I have an export or declare statement in my .bashrc file, like

```bash

declare-xPYTHONPATH=~/ASE/ML/itmal:$PYTHONPATH

```

but your ```itmal```, the [GITMAL] root dir, may be placed elsewhere.

For ___Windows___, you have to add `PYTHONPATH` to your user environment variables...see screenshot below (enlarge by modding the image width-tag or find the original png in the Figs directory).

<imgsrc="https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L01/Figs/Screenshot_windows_enviroment_variables.png"alt="WARNING: you need to be logged into Blackboard to view images"style="width:350px">

<imgsrc="https://itundervisning.ase.au.dk/GITMAL/L01/Figs/Screenshot_windows_enviroment_variables.png"alt="WARNING: you need to be logged into Blackboard to view images"style="width:350px">

or if you, like me, hate setting up things in a GUI, and prefer a console, try in a CMD on windows

replacing the username and path with whatever you have. If everything fails you could programmatically add your path to the libitmal directory as

```python

importsys,os

sys.path.append(os.path.expanduser('~/itmal'))

fromlibitmalimportutilsasitmalutils

print(dir(itmalutils))

print(itmalutils.__file__)

```

For the journal: remember to document your particular PATH setup.

%% Cell type:code id: tags:

``` python

# TODO: Qa...

```

%% Cell type:markdown id: tags:

#### Qb Create your own module, with some functions, and test it

Now create your own module, with some dummy functionality. Load it and run you dummy function in a Jupyter Notebook.

Keep this module at hand, when coding, and try to capture reusable python functions in it as you invent them!

For the journal: remember to document your particular library setup (where did you place files, etc).

%% Cell type:code id: tags:

``` python

# TODO: Qb...

```

%% Cell type:markdown id: tags:

#### Qc How do you 'recompile' a module?

When changing the module code, Jupyter will keep running on the old module. How do you force the Jupyter notebook to re-load the module changes?

%% Cell type:code id: tags:

``` python

# TODO: Qc...

```

%% Cell type:markdown id: tags:

#### [OPTIONAL] Qd Write a Howto on Python Modules a Packages

Write a short description of how to use modules in Python (notes on modules path, import directives, directory structure, etc.)

%% Cell type:code id: tags:

``` python

# TODO: Qd...

```

%% Cell type:markdown id: tags:

### Classes in Python

Good news: Python got classes. Bad news: they are somewhat obscure compared to C++ classes.

Though we will not use object-oriented programming in Python intensively, we still need some basic understanding of Python classes. Let's just dig into a class-demo, here is `MyClass` in Python

```python

classMyClass:

myvar="blah"

defmyfun(self):

print("This is a message inside the class.")

myobjectx=MyClass()

```

NOTE: The following exercise assumes some C++ knowledge, in particular the OPRG and OOP courses. If you are an EE-student, then ignore the cryptic C++ comments, and jump directly to some Python code instead. It's the Python solution here, that is important!

#### Qe Extend the class with some public and private functions and member variables

How are private function and member variables represented in python classes?

What is the meaning of `self` in python classes?

What happens to a function inside a class if you forget `self` in the parameter list, like `def myfun():` instead of `def myfun(self):` and you try to call it like `myobjectx.myfun()`? Remember to document the demo code and result.

[OPTIONAL] What does 'class' and 'instance variables' in python correspond to in C++? Maybe you can figure it out, I did not really get it reading, say this tutorial

Figure a way to declare/define a constructor (CTOR) in a python class. How is it done in python?

Is there a class destructor in python (DTOR)? Give a textual reason why/why-not python has a DTOR?

Hint: python is garbage collection like in C#, and do not go into the details of `__del__, ___enter__, __exit__` functions...unless you find it irresistible to investigate.

%% Cell type:code id: tags:

``` python

# TODO: Qf...

```

%% Cell type:markdown id: tags:

#### Qg Extend the class with a to-string function

Then find a way to serialize a class, that is to make some `tostring()` functionality similar to a C++

If you do not know C++, you might be aware of the C# way to string serialize

```

string s=myobject.tostring()

```

that is a per-class buildin function `tostring()`, now what is the pythonic way of 'printing' a class instance?

%% Cell type:code id: tags:

``` python

# TODO: Qg...

```

%% Cell type:markdown id: tags:

#### [OPTIONAL] Qh Write a Howto on Python Classes

Write a _How-To use Classes Pythonically_, including a description of public/privacy, constructors/destructors, the meaning of `self`, and inheritance.

"Remember the data-flow model for supervised learning\n",

"\n",

"<img src=\"https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L02/Figs/ml_simple_vector.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:500px\">\n",

"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L02/Figs/ml_simple_vector.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:500px\">\n",

"\n",

"Let us now express $J$ in terms of vectors and matrices instead of summing over individual scalars, and let's use $\\norm{2}$ as the distance function\n",

"\n",

...

...

@@ -464,25 +464,26 @@

"source": [

"REVISIONS||\n",

"---------||\n",

"2018-1218| CEF, initial. \n",

"2019-0131| CEF, spell checked and update. \n",

"2019-0204| CEF, changed d1/d2 in Qb to L1/L2. Fixe rev date error.\n",

"2019-0204| CEF, changed headline.\n",

"2019-0204| CEF, changed (.) in dist(x,y) to use pipes instead.\n",

"2019-0204| CEF, updated supervised learning fig, and changed , to ; for thetas, and change = to propto.\n",

"2019-0205| CEF, post lesson update, minor changes, added fabs around two test vectors.\n",

"2019-0207| CEF, updated def section. \n",

"2019-0901| CEF, updated for ITMAL v2.\n",

"2019-0904| CEF, updated for print-f and added conclusion Q.\n",

"2019-0905| CEF, fixed defect in print string and commented on fabs.\n",

"2020-0130| CEF, F20 ITMAL update.\n",

"2020-0203| CEF, minor text fixes.\n",

"2020-0224| CEF, elaborated on MAE and RMSE, emphasized not to use np functionality in L1 and L2.\n",

but sometimes the notation is a little more fuzzy, leaving out the transpose operator for $\mathbf x$ and in doing so just interpreting the $\mathbf{x}^{(i)}$'s to be row vectors instead of column vectors.

The target column vector, $\mathbf y$, also has the dimension $n$

$$

\by = \ac{c}{

y\pown{1} \\

y\pown{2} \\

\vdots \\

y\pownn \\

}

$$

#### Qa Given the following $\mathbf{x}^{(i)}$'s, construct and print the $\mathbf X$ matrix in python.

$$

\ar{rl}{

\bx\pown{1} &= \ac{c}{ 1, 2, 3}^T \\

\bx\pown{2} &= \ac{c}{ 4, 2, 1}^T \\

\bx\pown{3} &= \ac{c}{ 3, 8, 5}^T \\

\bx\pown{4} &= \ac{c}{-9,-1, 0}^T

}

$$

##### Implementation Details

Notice that the ```np.matrix``` class is getting deprecated! So, we use numpy's ```np.array``` as matrix container. Also, __do not__ use the built-in python lists or the numpy matrix subclass.

%% Cell type:code id: tags:

``` python

# Qa

importnumpyasnp

y=np.array([1,2,3,4])# NOTE: you'll need this later

# TODO..create and print the full matrix

assertFalse,"TODO: solve Qa, and remove me.."

```

%% Cell type:markdown id: tags:

### Norms, metrics or distances

The $\norm{2}$ Euclidian distance, or norm, for a vector of size $n$ is defined as

This Euclidian norm is sometimes also just denoted as $||\bx||$, leaving out the 2 in the subscript.

The squared $\norm{2}$ for a vector can compactly be expressed via

$$

\norm{2}^2: ||\bx||_2^2 = \bx^\top\bx

$$

by the general dot or inner-product (taking $\by=\bx$ in the $\norm{2}^2$ above)

$$

\ar{ll}{

\langle\bx,\by\rangle &= \bx\cdot\by\\

&=\bx^\top \by\\

&= \sum_{i=1}^n x_{i} y_{i} \\

&= ||\bx|| ~ ||\by|| \cos{\theta}

}

$$

taking $\theta$ to be zero, and hence $cos(\theta)=1$ when calculating $\langle\bx,\bx\rangle$

The $\norm{1}$ 'City-block' norm is given by

$$

\norm{1}:~~ ||\bx||_1 = \sum_i |x_i|

$$

but $\norm{1}$ is not used as intensive as its more popular $\norm{2}$ cousin.

Notice that $|x|$ in code means ```fabs(x)```.

#### Qb Implement the $\norm{1}$ and $\norm{2}$ norms for vectors in python.

First implementation must be a 'low-level'/explicit implementation---using primitive/build-in functions, like ```+```, ```*``` and power ```**``` only! The square-root function can be achieved via power like ```x**0.5```.

Do NOT use any methods from libraries, like ```math.sqrt```, ```math.abs```, ```numpy.linalg.inner```, ```numpy.dot()``` or similar. Yes, using such libraries is an efficient way of building python software, but in this exercise we want to explicitly map the mathematichal formulaes to python code.

Name your functions L1 and L2 respectively, they both take one vector as input argument.

But test your implementation against some built-in functions, say ```numpy.linalg.norm```

When this works, and passes the tests below, optimize the $\norm{2}$, such that it uses np.numpy's dot operator instead of an explicit sum, call this function L2Dot. This implementation must be pythonic, i.e. it must not contain explicit for- or while-loops.

%% Cell type:code id: tags:

``` python

# TODO: solve Qb...implement the L1, L2 and L2Dot functions...

assertFalse,"TODO: solve Qb, and remove me.."

# TEST vectors: here I test your implementation...calling your L1() and L2() functions

assertfabs(d1-expected_d1)<eps,"L1 dist seems to be wrong"

assertfabs(d2-expected_d2)<eps,"L2 dist seems to be wrong"

print("OK(part-1)")

# comment-in once your L2Dot fun is ready...

#d2dot=L2Dot(tx-ty)

#print("d2dot-expected_d2=",d2dot-expected_d2)

#assert fabs(d2dot-expected_d2)<eps, "L2Ddot dist seem to be wrong"

#print("OK(part-2)")

```

%% Cell type:markdown id: tags:

## The cost function, $J$

Now, most ML algorithm uses norms or metrics internally when doing minimizations. Details on this will come later, but for now we need to know that an algorithm typically tries to minimize a given performance metric, the loss function, for all the input data, and implicitly tries to minimize the sum of all norms for the 'distances' between some predicted output, $y\st{pred}$ and the true output $y\st{true}$, with the distance between these typically given by the $\norm{2}$ norm

### Cost function in vector/matrix notation using $\norm{2}$

Remember the data-flow model for supervised learning

<imgsrc="https://blackboard.au.dk/bbcswebdav/courses/BB-Cou-UUVA-94506/Fildeling/L02/Figs/ml_simple_vector.png"alt="WARNING: you need to be logged into Blackboard to view images"style="width:500px">

<imgsrc="https://itundervisning.ase.au.dk/GITMAL/L02/Figs/ml_simple_vector.png"alt="WARNING: you need to be logged into Blackboard to view images"style="width:500px">

Let us now express $J$ in terms of vectors and matrices instead of summing over individual scalars, and let's use $\norm{2}$ as the distance function

#### Loss or Objective Function using the Mean Squared Error

This formulation is equal to the definition of the _mean-squared-error_, MSE (or indirectly also RMSE), here given in the general formulation for some random variable $Z$

So, using the $\norm{2}$ for the distance metric, is equal to saying that we want to minimize $J$ with respect to the MSE

$$

\ar{rl}{

J &= \mbox{MSE}(h(\bX), \by\st{true}) \\

&= \mbox{MSE}(\by\st{pred}~, \by\st{true}) \\

&= \mbox{MSE}(\hat{\by}, \by\st{true})

}

$$

Note: when minimizing one can ignore the constant factor $1/n$ and it really does not matter if you minimize MSE or RMSE. Often $J$ is also multiplied by 1/2 to ease notation when trying to differentiate it.

Now, let us take a look on how you calculate the MSE.

The MSE uses the $\norm{2}$ norm internally, well, actually $||\cdot||^2_2$ to be precise, and basically just sums, means and roots the individual (scalar) losses (distances), we just saw before.

And the RMSE is just an MSE with a final square-root call.

### Qc Construct the Root Mean Square Error (RMSE) function (Equation 2-1 [HOML]).

Call the function RMSE, and evaluate it using the $\bX$ matrix and $\by$ from Qa.

We implement a dummy hypothesis function, that just takes the first column of $\bX$ as its 'prediction'

$$

h\st{dummy}(\bX) = \bX(:,0)

$$

Do not re-implement the $\norm{2}$ for the RMSE function, but call the '''L2''' function you just implemented internally in RMSE.

%% Cell type:code id: tags:

``` python

# TODO: solve Qc...implement your RMSE function here

assertFalse,"TODO: solve Qc, and remove me.."

# Dummy h function:

defh(X):

ifX.ndim!=2:

raiseValueError("excpeted X to be of ndim=2, got ndim=",X.ndim)

ifX.shape[0]==0orX.shape[1]==0:

raiseValueError("X got zero data along the 0/1 axis, cannot continue")

returnX[:,0]

# Calls your RMSE() function:

r=RMSE(h(X),y)

# TEST vector:

eps=1E-9

expected=6.57647321898295

print(f"RMSE={r}, diff={r-expected}")

assertfabs(r-expected)<eps,"your RMSE dist seems to be wrong"

print("OK")

```

%% Cell type:markdown id: tags:

### MAE

#### Qd Similar construct the Mean Absolute Error (MAE) function (Equation 2-2 [HOML]) and evaluate it.

The MAE will algorithmic wise be similar to the MSE part from using the $\norm{1}$ instead of the $\norm{2}$ norm.

Again, re-implementation of the$\norm{1}$ is a no-go, call the '''L1''' instead internally i MAE.

%% Cell type:code id: tags:

``` python

# TODO: solve Qd

assertFalse,"TODO: solve Qd, and remove me.."

# Calls your MAE function:

r=MAE(h(X),y)

# TEST vector:

expected=3.75

print(f"MAE={r}, diff={r-expected}")

assertfabs(r-expected)<eps,"MAE dist seems to be wrong"

print("OK")

```

%% Cell type:markdown id: tags:

## Pythonic Code

### Robustness of Code

Data validity checking is an essential part of robust code, and in Python the 'fail-fast' method is used extensively: instead of lingering on trying to get the 'best' out of an erroneous situation, the fail-fast pragma will be very loud about any data inconsistencies at the earliest possible moment.

Hence robust code should include a lot of error checking, say as pre- and post-conditions (part of the design-by-contract programming) when calling a function: when entering the function you check that all parameters are ok (pre-condition), and when leaving you check the return parameter (post-conditions).

Normally assert-checking or exception-throwing will do the trick just fine, with the exception method being more _pythonic_.

For the norm-function you could, for instance, test your input data to be 'vector' like, i.e. like

```python

assertx.shape[0]>=0andx.shape[1]==0

ifnotx.ndim==1:

raisesomeerror

```

or similar.

#### Qe Robust Code

Add error checking code (asserts or exceptions), that checks for right $\hat\by$-$\by$ sizes of the MSE and MAE functions.

Also add error checking to all you previously tested L2() and L1() functions, and re-run all your tests.

%% Cell type:code id: tags:

``` python

# TODO: solve Qe...you need to modify your python cells above

assertFalse,"TODO: solve Qe, and remove me.."

```

%% Cell type:markdown id: tags:

### Qf Conclusion

Now, conclude on all the exercise above.

Write a short textual conclusion (max. 10- to 20-lines) that extract the _essence_ of the exercises: why did you think it was important to look at these particular ML concepts, and what was our overall learning outcome of the exercises (in broad terms).