Skip to content

GitLab

Commit 8204c0e1 authored by Carsten Eie Frigaard's avatar Carsten Eie Frigaard
Browse files

added_new_lesson07_NNs_II

parent a74b8f48
Hide whitespace changes
Inline Side-by-side
L07/installing_keras.ipynb 0 → 100644
View file @ 8204c0e1
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# ITMAL Demo\n",
"\n",
"## Installing Keras\n",
"\n",
"REVISIONS| |\n",
"---------| |\n",
"2018-0325| CEF, initial.\n",
"2020-0305| CEF, F20 ITMAL update.\n",
"2020-0306| CEF, investigated Anaconda 2019.10 on Windows and updated GPU server notes.\n",
"2021-1012| CEF, updated for ITMAL E21.\n",
"\n",
"1: Launch the __anaconda prompt__ console (CMD), via the Start menu\n",
"\n",
"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:200px\">\n",
"\n",
"\n",
"2: list installed packages via \n",
"\n",
"```bash\n",
"> conda list\n",
"```\n",
"\n",
"in the anaconda console.\n",
"\n",
"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt_install_0.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:700px\">\n",
"\n",
"\n",
"3: install keras via\n",
"\n",
"```bash\n",
"> conda install keras\n",
"```\n",
"\n",
"and WAIT for 1 to 30 min before the spin-progress bar finish (a problem makes `conda` extreme slow in the latest two releases of anaconda!).\n",
"\n",
"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt_install_1.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:700px\">\n",
"\n",
"\n",
"After install, you can see the Keras and Tensorflow version via ```conda list keras``` and ```conda list tensorflow```, but notice that you might also want to install the GPU version of Tensoflow, if our PC has a suitable GPU (need CUDA support). Below I did not install the GPU version seen by the call ```conda list tensorflow-gpu```\n",
"\n",
"<img src=\"https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt_install_2.png\" alt=\"WARNING: you need to be logged into Blackboard to view images\" style=\"width:700px\">\n",
"\n",
"4: if it downgrades your Scikit-learn (use version function in the cell below), then try removing keras and/or tensorflow and reinstall\n",
"\n",
"```bash\n",
"> conda remove keras tensorflow\n",
"```\n",
"\n",
"```bash\n",
"> conda install keras tensorflow\n",
"```\n",
"\n",
"or perhaps try installing from conda-forge\n",
"\n",
"```\n",
"conda install -c conda-forge tensorflow keras \n",
"```\n",
"\n",
"5: if everything fails: use the ASE GPU cluster or use keras in TensorFlow ala\n",
"\n",
"```python\n",
"import tensorflow as tf\n",
"mnist = tf.keras.datasets.mnist.load_data()\n",
"```\n",
"\n",
"My local installation has the following version setup (yours may vary)\n",
"\n",
"Initial:\n",
"\n",
"```python\n",
"Python version: 3.7.4.\n",
"Scikit-learn version: 0.21.3.\n",
"WARN: could not find keras!\n",
"WARN: could not find tensorflow!\n",
"WARN: could not find tensorflow.keras!\n",
"```\n",
"\n",
"and after installing Keras (and hence implicitly TensorFlow) on Windows\n",
"\n",
"```python\n",
"Python version: 3.7.4.\n",
"Scikit-learn version: 0.21.3.\n",
" Using TensorFlow backend.\n",
"Keras version: 2.3.1\n",
"Tensorflow version: 2.1.0\n",
"Tensorflow.keras version: 2.2.4-tf\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Python version: 3.7.4.\n",
"Scikit-learn version: 0.21.3.\n",
"Keras version: 2.2.4\n",
"Tensorflow version: 2.0.0\n",
"Tensorflow.keras version: 2.2.4-tf\n"
]
}
],
"source": [
"# DEMO of Versions in libitmal\n",
"from libitmal import versions as itmalversions\n",
"itmalversions.Versions()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using the ASE GPU Cluster"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Client GPU Support\n",
"\n",
"If your own computer has a CUDA-compatible GPU you might also want to install TensorFlow for the GPU\n",
"```\n",
"conda install tensorflow-gpu \n",
"```\n",
"\n",
"### Server GPU support\n",
"\n",
"You also have an ITMAL group account on our GPU Cluster server at\n",
"\n",
"* http://gpucluster.st.lab.au.dk/\n",
"\n",
"Find login details etc. in Blackboard (\"Kursusinfo | GPU Cluster\"):\n",
"\n",
"* https://blackboard.au.dk/webapps/blackboard/content/listContentEditable.jsp?content_id=_2485142_1&course_id=_134254_1#gpucluster\n",
"\n",
"Current GPU-Cluster version setup is\n",
"```python\n",
"Python version: 3.6.8.\n",
"Scikit-learn version: 0.20.3.\n",
"Keras version: 2.2.4\n",
"Tensorflow version: 1.12.0\n",
"```\n",
"\n",
"### Issues regarding the Server GPU Memory \n",
"\n",
"For all users, I've added a startup-script when you log into the GPU server. The startup-script is found in \n",
"\n",
"* /home/shared/00_init.py\n",
"\n",
"and among other things, add your home-folder to the PYTHON path.\n",
"\n",
"When running on the GPU-server you are automatically assigned 10% of the GPU memory. This is also done via the startup-script, and you are allowed to increase you GPU memory fraction if needed by calling the Enable GPU function in ```/home/shared/00_init.py``` (or the module ```kernelfun``` in ``libitmal```) like\n",
"```\n",
"StartupSequence_EnableGPU(gpu_mem_fraction=0.3, gpus=1, verbose=1)\n",
"``` \n",
"\n",
"or \n",
"\n",
"and thereby allocating 30% GPU mem. \n",
"\n",
"NOTE 1: processes using more than 50% GPU memory will automatically be killed with an interval of about 5 min, and python kernels running for more than about a week will also be terminated automatically.\n",
"\n",
"NOTE 2: most Scikit-learn ML algorithms (if not all) do NOT use the GPU at all. You need to move to Tensorflow/Keras to get true GPU hardware support.\n",
"\n",
"NOTE 3: notebooks will keep running on the server, even if you shut-down you web connection to it. Print output will hence be lost, but you can still start long running model training on the server, and come back later to see if its finished...(on the same Node).\n",
"\n",
"NOTE 4: If you need to stop you server: use the \"Control Panel (upper right) | Stop my server\" to shut down all your kernels and release all memory."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ERROR: something failed in EnableGPU(), Tensorflow part\n",
"/bin/sh: 1: nvidia-smi: not found\r\n"
]
}
],
"source": [
"# DEMO of set GPU memory fraction in libitmal\n",
"from libitmal import kernelfuns as itmalkernefuns\n",
"itmalkernefuns.EnableGPU()\n",
"\n",
"# See kernel running, but works only if you got CUDA installed\n",
"! nvidia-smi"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### GPU Server GIT/PYTHONPATH setup\n",
"\n",
"On the GPU server You can clone the git repository inside the Jupyternotbook via\n",
"```bash\n",
"! git clone https://cfrigaard@bitbucket.org/cfrigaard/itmal\n",
"```\n",
"\n",
"The `PYTHONPATH` environment should already point to you home folder via the startup-script (described above)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": true
}
},
"nbformat": 4,
"nbformat_minor": 2
}
%% Cell type:markdown id: tags:
# ITMAL Demo
## Installing Keras
REVISIONS| |
---------| |
2018-0325| CEF, initial.
2020-0305| CEF, F20 ITMAL update.
2020-0306| CEF, investigated Anaconda 2019.10 on Windows and updated GPU server notes.
2021-1012| CEF, updated for ITMAL E21.
1: Launch the __anaconda prompt__ console (CMD), via the Start menu
<img src="https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt.png" alt="WARNING: you need to be logged into Blackboard to view images" style="width:200px">
2: list installed packages via
```bash
> conda list
```
in the anaconda console.
<img src="https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt_install_0.png" alt="WARNING: you need to be logged into Blackboard to view images" style="width:700px">
3: install keras via
```bash
> conda install keras
```
and WAIT for 1 to 30 min before the spin-progress bar finish (a problem makes `conda` extreme slow in the latest two releases of anaconda!).
<img src="https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt_install_1.png" alt="WARNING: you need to be logged into Blackboard to view images" style="width:700px">
After install, you can see the Keras and Tensorflow version via ```conda list keras``` and ```conda list tensorflow```, but notice that you might also want to install the GPU version of Tensoflow, if our PC has a suitable GPU (need CUDA support). Below I did not install the GPU version seen by the call ```conda list tensorflow-gpu```
<img src="https://itundervisning.ase.au.dk/GITMAL/L07/Figs/Screenshot_anaconda_prompt_install_2.png" alt="WARNING: you need to be logged into Blackboard to view images" style="width:700px">
4: if it downgrades your Scikit-learn (use version function in the cell below), then try removing keras and/or tensorflow and reinstall
```bash
> conda remove keras tensorflow
```
```bash
> conda install keras tensorflow
```
or perhaps try installing from conda-forge
```
conda install -c conda-forge tensorflow keras
```
5: if everything fails: use the ASE GPU cluster or use keras in TensorFlow ala
```python
import tensorflow as tf
mnist = tf.keras.datasets.mnist.load_data()
```
My local installation has the following version setup (yours may vary)
Initial:
```python
Python version: 3.7.4.
Scikit-learn version: 0.21.3.
WARN: could not find keras!
WARN: could not find tensorflow!
WARN: could not find tensorflow.keras!
```
and after installing Keras (and hence implicitly TensorFlow) on Windows
```python
Python version: 3.7.4.
Scikit-learn version: 0.21.3.
Using TensorFlow backend.
Keras version: 2.3.1
Tensorflow version: 2.1.0
Tensorflow.keras version: 2.2.4-tf
```
%% Cell type:code id: tags:
``` python
# DEMO of Versions in libitmal
from libitmal import versions as itmalversions
itmalversions.Versions()
```
%%%% Output: stream
Python version: 3.7.4.
Scikit-learn version: 0.21.3.
Keras version: 2.2.4
Tensorflow version: 2.0.0
Tensorflow.keras version: 2.2.4-tf
%% Cell type:markdown id: tags:
# Using the ASE GPU Cluster
%% Cell type:markdown id: tags:
### Client GPU Support
If your own computer has a CUDA-compatible GPU you might also want to install TensorFlow for the GPU
```
conda install tensorflow-gpu
```
### Server GPU support
You also have an ITMAL group account on our GPU Cluster server at
* http://gpucluster.st.lab.au.dk/
Find login details etc. in Blackboard ("Kursusinfo | GPU Cluster"):
* https://blackboard.au.dk/webapps/blackboard/content/listContentEditable.jsp?content_id=_2485142_1&course_id=_134254_1#gpucluster
Current GPU-Cluster version setup is
```python
Python version: 3.6.8.
Scikit-learn version: 0.20.3.
Keras version: 2.2.4
Tensorflow version: 1.12.0
```
### Issues regarding the Server GPU Memory
For all users, I've added a startup-script when you log into the GPU server. The startup-script is found in
* /home/shared/00_init.py
and among other things, add your home-folder to the PYTHON path.
When running on the GPU-server you are automatically assigned 10% of the GPU memory. This is also done via the startup-script, and you are allowed to increase you GPU memory fraction if needed by calling the Enable GPU function in ```/home/shared/00_init.py``` (or the module ```kernelfun``` in ``libitmal```) like
```
StartupSequence_EnableGPU(gpu_mem_fraction=0.3, gpus=1, verbose=1)
```
or
and thereby allocating 30% GPU mem.
NOTE 1: processes using more than 50% GPU memory will automatically be killed with an interval of about 5 min, and python kernels running for more than about a week will also be terminated automatically.
NOTE 2: most Scikit-learn ML algorithms (if not all) do NOT use the GPU at all. You need to move to Tensorflow/Keras to get true GPU hardware support.
NOTE 3: notebooks will keep running on the server, even if you shut-down you web connection to it. Print output will hence be lost, but you can still start long running model training on the server, and come back later to see if its finished...(on the same Node).
NOTE 4: If you need to stop you server: use the "Control Panel (upper right) | Stop my server" to shut down all your kernels and release all memory.
%% Cell type:code id: tags:
``` python
# DEMO of set GPU memory fraction in libitmal
from libitmal import kernelfuns as itmalkernefuns
itmalkernefuns.EnableGPU()
# See kernel running, but works only if you got CUDA installed
! nvidia-smi
```
%%%% Output: stream
ERROR: something failed in EnableGPU(), Tensorflow part
/bin/sh: 1: nvidia-smi: not found
%% Cell type:markdown id: tags:
### GPU Server GIT/PYTHONPATH setup
On the GPU server You can clone the git repository inside the Jupyternotbook via
```bash
! git clone https://cfrigaard@bitbucket.org/cfrigaard/itmal
```
The `PYTHONPATH` environment should already point to you home folder via the startup-script (described above).
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment