Commit f787be98 authored by Carsten Eie Frigaard's avatar Carsten Eie Frigaard
Browse files

pre_l08_update

parent 16cd2d1d
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# ITMAL Demo # ITMAL Demo
## Installing Keras ## Installing Keras
REVISIONS| | REVISIONS| |
---------| | ---------| |
2018-03-25| CEF, initial. 2018-03-25| CEF, initial.
2020-03-05| CEF, F20 ITMAL update. 2020-03-05| CEF, F20 ITMAL update.
2020-03-06| CEF, investigated Anaconda 2019.10 on Windows and updated GPU server notes. 2020-03-06| CEF, investigated Anaconda 2019.10 on Windows and updated GPU server notes.
2021-10-12| CEF, updated for ITMAL E21. 2021-10-12| CEF, updated for ITMAL E21.
2021-03-23| CEF, updated for SWMAL F22. 2021-03-23| CEF, updated for SWMAL F22, rewrote install method for Keras via environments.
### WARNING for SWMAL F22 ### Installing Keras and Tensorflow for Anaconda 2021.11
Keras will not install under Anaconda. It ends up in a endless package-conflict, when installing under `conda`. Keras will not install under Anaconda version 2021.11. It ends up in a endless package-conflict, when installing under `conda`.
Fix is underway.. (This is a new finding for the particular version 2021.11, and previous version did not have this tensorflow install problem...but typically only a Keras install problem).
The root-cause of the problem must be missing check when packageing the Anaconda, failing to find the set of conflicts we see, when installing `tensorflow` in the set of default installe packages that comes with the Anaconda distribution.
### Install Keras via Anaconda Prompt #### 1) Prepare and Ceate a new Environment
1: Launch the __anaconda prompt__ console (CMD), via the Start menu So, one solution is to create a new conda environment, and from this install `scikit-learn` and `tensorflow` and `keras`,
<img src="https://itundervisning.ase.au.dk/SWMAL/L06/Figs/Screenshot_anaconda_prompt.png" alt="WARNING: could not get image from server." style="width:200px"> Later we need a package called `nb_conda_kernels`, let us install this before we create and activate the new enviroment
2: list installed packages via
```bash ```bash
> conda list > conda install nb_conda_kernels
``` ```
in the anaconda console. Now, let us call our enviroment `swmal` and create it by running
<img src="https://itundervisning.ase.au.dk/SWMAL/L06/Figs/Screenshot_anaconda_prompt_install_0.png" alt="WARNING: could not get image from server." style="width:700px">
3: install keras via
```bash ```bash
> conda install keras > conda create --name swmal
``` ```
and WAIT for 1 to 30 min before the spin-progress bar finish (a problem that makes `conda` extreme slow in the latest two releases of anaconda!). and then activate it via
<img src="https://itundervisning.ase.au.dk/SWMAL/L06/Figs/Screenshot_anaconda_prompt_install_1.png" alt="WARNING: could not get image from server." style="width:700px">
After install, you can see the Keras and Tensorflow version via ```conda list keras``` and ```conda list tensorflow```, but notice that you might also want to install the GPU version of Tensoflow, if our PC has a suitable GPU (need CUDA support). Below I did not install the GPU version seen by the call ```conda list tensorflow-gpu```
<img src="https://itundervisning.ase.au.dk/SWTMAL/L06/Figs/Screenshot_anaconda_prompt_install_2.png" alt="WARNING: could not get image from server." style="width:700px">
4: if it downgrades your Scikit-learn (use version function in the cell below), then try removing keras and/or tensorflow and reinstall
```bash ```bash
> conda remove keras tensorflow > conda activate swmal
``` ```
```bash #### 2) Install Needed Packages
> conda install keras tensorflow
```
or perhaps try installing from conda-forge Now we got a clean state enviroment and we need to install the packages needed for `scikit-learn` and `tensorflow`, but this is as easy as
```bash
> conda install scikit-learn tensorflow keras nb_conda_kernels
``` ```
conda install -c conda-forge tensorflow keras
```
5: if everything fails: use the ASE GPU cluster or use keras in TensorFlow ala
```python
import tensorflow as tf
mnist = tf.keras.datasets.mnist.load_data()
```
My local installation has the following version setup (yours may vary)
Initial:
```python
Python version: 3.8.5.
Scikit-learn version: 0.23.2.
WARN: could not find keras!
WARN: could not find tensorflow!
WARN: could not find tensorflow.keras!
```
and after installing Keras (and hence implicitly TensorFlow) on Windows
```python
Python version: 3.8.5.
Scikit-learn version: 0.23.2.
Keras version: 2.4.3
Tensorflow version: 2.2.0
Tensorflow.keras version: 2.3.0-tf
Opencv2 version: 4.5.1
```
### Alternative 1: Installing Keras via Tensorflow
If Keras and TensorFlow (TF) start a battle-of-versions (Keras wants one version TF another, it happens frequently) you could also go for just use the Keras already found in TF. (Notice that the package `nb_conda_kernels` also needs to be installed in this new enviroment.)
So, yes, there is already a Keras in the TF interface that can be used directly as #### 3) Launching Jupyter-Notebooks in the new Enviroment
```tf.keras.(some modules or functions)``` There are two `keras` interfaces, the stand-alone `keras.` and then a similar interface already build into `tensorflow.keras`, you could use both, but direct access via the stand-alone inteface may be the most obvious.
instead of direct keras interface calls via
```keras.(some modules or functions)``` #### 4) Testing the New Enviroment Setup
### Alternative 2: Install Keras via Anaconda GUI Lets see the version installed in the new `swmal` environment via the `Versions()` function found in the `itmallib`
If you dislike the Anacoda prompt and prefer a GUI, then launch the Anaconda Navigator, go to the Environment tab and enter 'keras' in the search prompt
<img src="https://itundervisning.ase.au.dk/SWMAL/L06/Figs/Screenshot_anaconda_prompt_install_3.png" alt="WARNING: could not get image from server." style="width:700px">
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# DEMO of Versions in libitmal from libitmal import versions
from libitmal import versions as itmalversions versions.Versions()
itmalversions.Versions()
``` ```
%% Output %% Output
Python version: 3.9.7. Python version: 3.9.7.
Scikit-learn version: 0.24.2. Scikit-learn version: 1.0.2.
WARN: could not find keras! Keras version: 2.6.0
WARN: could not find tensorflow! Tensorflow version: 2.6.0
WARN: could not find tensorflow.keras! Tensorflow.keras version: 2.6.0
Opencv2 version: 4.5.5 Opencv2 version: 4.5.5
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Using the ASE GPU Cluster The `Versions()` function should print whatever version you installed, or produce a warning, if the package is not installed at all.
__NOTE: this section is currently slighty outdated!__ For my current Windows/Anaconda setup I got the versions, Yours may differ slightly
### Client GPU Support
If your own computer has a CUDA-compatible GPU you might also want to install TensorFlow for the GPU
``` ```
conda install tensorflow-gpu Python version: 3.9.7.
Scikit-learn version: 1.0.2.
Keras version: 2.6.0
Tensorflow version: 2.6.0
Tensorflow.keras version: 2.6.0
``` ```
### Server GPU support #### 5) Wrapping It All Up in a BAT File
You also have an ITMAL group account on our GPU Cluster server at
* http://gpucluster.st.lab.au.dk/ To make development easy, a BAT (Windows batch or script file) shold be created. This should ease the launch of Jupyter-Notebooks and the BAT file could be put in a icon placed on the taskbar or similar.
Find login details etc. in Blackboard ("Kursusinfo | GPU Cluster"): The BAT-file should contain the followin text lines, an notice that you must changex
* https://brightspace.au.dk/d2l/le/lessons/27524/topics/296678
Current GPU-Cluster version setup is (??)
```python
Python version: 3.6.8.
Scikit-learn version: 0.20.3.
Keras version: 2.2.4
Tensorflow version: 1.12.0
```
### Issues regarding the Server GPU Memory
For all users, I've added a startup-script when you log into the GPU server. The startup-script is found in
* /home/shared/00_init.py
and among other things, add your home-folder to the PYTHON path.
When running on the GPU-server you are automatically assigned 10% of the GPU memory. This is also done via the startup-script, and you are allowed to increase you GPU memory fraction if needed by calling the Enable GPU function in ```/home/shared/00_init.py``` (or the module ```kernelfun``` in ``libitmal```) like
```
StartupSequence_GPU(verbose=True)
``` ```
ECHO OFF
or REM my-jupyter-notebook
REM Version: 0.1
REM 2022-03-23: CEF, inital version
and thereby allocating 30% GPU mem. echo MY-JUPYTER-NOTEBOOK launcher..
NOTE 1: processes using more than 50% GPU memory will automatically be killed with an interval of about 5 min, and python kernels running for more than about a week will also be terminated automatically. SET USER=au204573
NOTE 2: most Scikit-learn ML algorithms (if not all) do NOT use the GPU at all. You need to move to Tensorflow/Keras to get true GPU hardware support. @CALL "%C:\Users\%USER%\Anaconda3\condabin\conda.bat" activate swmal %*
NOTE 3: notebooks will keep running on the server, even if you shut-down you web connection to it. Print output will hence be lost, but you can still start long running model training on the server, and come back later to see if its finished...(on the same Node). REM note book start in this directory, you may change it:
cd \
NOTE 4: If you need to stop you server: use the "Control Panel (upper right) | Stop my server" to shut down all your kernels and release all memory. jupyter-notebook
%% Cell type:code id: tags: echo DONE
``` python
# DEMO of set GPU memory fraction in libitmal
from libitmal import kernelfuns as itmalkernefuns
itmalkernefuns.StartupSequence_GPU(verbose=True)
# See kernel running, but works only if you got CUDA installed
! nvidia-smi
```
%% Cell type:markdown id: tags:
### GPU Server GIT/PYTHONPATH setup
On the GPU server You can clone the git repository inside the Jupyternotbook via
```bash
! git clone https://gitlab.au.dk/au204573/GITMAL.git
! cd GITMAL && git pull
``` ```
The `PYTHONPATH` environment should already point to you home folder via the startup-script (described above).
......
ECHO OFF
REM my-jupyter-notebook
REM Version: 0.1
REM 2022-03-23: CEF, inital version
echo MY-JUPYTER-NOTEBOOK launcher..
@CALL "%HOMEPATH%\Anaconda3\condabin\conda.bat" activate swmal %* <NUL
REM notebooks start in this directory, you may change it..
cd \
REM then launch the notebook..
jupyter-notebook
echo DONE
\ No newline at end of file