bomonike

“Notebooks” display HTML markup as comments between executable Python code (for Machine Learning and Data Science), for Style Transfer of images, etc.

Overview

Google Colab (Colaboratory)
Browser configuration on your laptop
- Jupyter notebook and session files
- Ignore checkpoints on push
macOS Server Installation
Server Installation
Run Notebook UI
Turi (Dato) Python algorithms
References
- Additional
More about Python

The name “Jupyter” comes from the combination of languages Julia, Python, and R (the statistics package).

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Jupyter is a web application server that runs several separate environments (one on each port). It’s open-sourced at:

https://github.com/jupyterlab/jupyterlab.git

Each Jupyter notebook contains explanatory text, math equations, code, and visualizations all in one easily sharable document.

Jupyter Notebooks are JSON files with the extension .ipynb. However, “.txt” is added for storage.

Notebooks used to be called “iPython” notebooks before languages other than Python were added.

Below is a guided learning experience. You perform actions in a planned sequence that takes you through various features. “PROTIP” tags commentary to highlight ideas probably unique to this website.ß

Google Colab (Colaboratory)

Google provides a FREE on-line environment with zero configuration:

https://colab.research.google.com

python-colab-welcome-1436x1216

In the above example, data is received from the numpy (np) library’s random function, rounded down.

Photo Style Transfer

There are other tools that do this. But as a practical example of learning about Colab for programming Python Machine Learning, let’s do a “style transfer” on a photo file you provide:

Click on this link to open the “TF-Hub: Fast Style Transfer for Arbitrary Styles.ipynb” within Colab:

https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf2_arbitrary_image_stylization.ipynb

import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
import numpy as np
import PIL.Image
&nbsp;
def load_img(path_to_img):
max_dim = 512
img = tf.io.read_file(path_to_img)
img = tf.image.decode_image(img, channels=3)
img = tf.image.convert_image_dtype(img, tf.float32)
&nbsp;
shape = tf.cast(tf.shape(img)[:-1], tf.float32)
long_dim = max(shape)
scale = max_dim / long_dim
&nbsp;
new_shape = tf.cast(shape * scale, tf.int32)
&nbsp;
img = tf.image.resize(img, new_shape)
img = img[tf.newaxis, :]
return img
&nbsp;
def imshow(image, title=None):
if len(image.shape) > 3:
   image = tf.squeeze(image, axis=0)
&nbsp;
plt.imshow(image)
if title:
   plt.title(title)
&nbsp;
def tensor_to_image(tensor):
tensor = tensor*255
tensor = np.array(tensor, dtype=np.uint8)
if np.ndim(tensor)>3:
   assert tensor.shape[0] == 1
   tensor = tensor[0]
return PIL.Image.fromarray(tensor)

To Run all Notebook cells: on a Mac, hold down command + fn to press the “F9”.

For a reminder of that keyboard sequence, click the “Runtime” menu.
Press (command+down arrow) to scroll down to the bottom to see the default original, style image, and stylized image:
Scroll down past the list of style images and click on each URL to view them.

The program creates a stylized image for each entry in the “content_url” dictionary.
Click the content_name: sea_turtle to select another original image.
Click the style_name: munch_scream to select another style image.

Style Transfer your picture’s URL
To run the code block:
- click on the rounded arrow or
- Press the Tab key then hold down Shift key to press Return.
At the left menu, click “To Run: Load more images” to see where you can add a statement pointing to your own photo.
Click on this URL, to go to many websites go to get a URL of a photo based on an email:

en.gravatar.com
Sign in using your WordPress email and password, or sign up and download your photo.
Right-click on the picture you want to use and select Copy image address.
Click on the “TF-Hub” tab to return to the Colab.

Replace the “grace_hopper” entry with the URL to your picture, for example:

grace_hopper='https://en.gravatar.com/userimage/10217204/6f0736298461c359b864b3939db37f8f.jpg',

To Run all Notebook cells: on a Mac, hold down command + fn to press the “F9”.
Select a different style_name and rerun to see a new stylized image.
When you see a stylized image you like, right-click on the picture and Save Image As… download.jpg in your Downloads folder:

Admittedly, the pictures are not high resolution.
Now edit that picture and share it.

Sample Notebook files in Colab

Navigate to a URL containing Jupyter Notebook files:

Sample Notebook code with “Open in Colab” are in Noah Gift’s “Minimal Python” book: https://github.com/paiml/minimal-python/blob/master/Chapter1_Minimal_Python.ipynb
https://github.com/jakevdp/PythonDataScienceHandbook
https://www.eecis.udel.edu/~boncelet/ipython.html
https://github.com/odewahn/ipynb-examples
https://github.com/ogrisel/notebooks
analysis of gravitational waves from two colliding blackholes detected by the LIGO experiment.
Sentiment-network.zip from Andrew Trask contains notebooks for sentiment analysis.

Browser configuration on your laptop

For possible offline access (on a cruise ship, a railcar through the wilderness, at the beach, etc.):

PROTIP: Set Chrome as your default browser.

REMEMBER: Jupyter’s Command Palette does not work in Firefox and Internet Explorer, only in Chrome and Apple Safari.

Jupyter notebook and session files
Make a container folder (directory) to hold Jupyter session files.

This is done automatically by doing a git clone.
Download a Jupyter Notebook file into your folder.

You may have to move downloaded files from your Downloads folder.

Ignore checkpoints on push
Edit the .gitignore file to add .ipynb_checkpoints/ so that such folders created automatically are not uploaded into the GitHub repository.
run a notebook.

macOS Server Installation

VIDEO: Without Python Anacoda:

https://www.youtube.com/watch?v=z7zOkRubIrU

Bring up Terminal
Install Python
python3 -V
pip3 -V
Create a folder to hold a Python virtual enviornment
```
cd
mkdir ~/jupyter-env
cd venv
python3 -m venv ~/jupyter-env
cd jupyter-env
ls  -ltr
```
File pyenv.cfg and directories lib, include, bin.
Activate the virtual environment so “(venv)” appears before the CLI prompt:
```
source ~/jupyter-env/bin/activate
```
Install Jupyter FIXME: This environment is externally managed
```
python3 -m pip install --upgrade pip
pip install jupyter
```
Verify what modules are installed (numpy, PyQt6, etc.):
```
python3 -m pip freeze
```
Start Jupyter server:
```
jupyter notebook
```

Server Installation

The canonical https://jupyter.org/install and Installation Guide says JupyterLab can be manually installed using conda or pip, described below.

However, you will find it easier to use the Docker image defined at https://github.com/jupyter/docker-stacks. See https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html

Even better, the easiest approach is to use my Bash script at bash/bash-sample.sh

https://github.com/wilsonmar/DevSecOps

Manual install

On a Mac: if you use pipenv, install using that. Otherwise:
```
pip3 install jupyterlab
```
This adds a bunch of “jupyter” files in the /bin folder.
Edit ~/.bash_profile to make sure the /bin folder is in the PATH.
```
export PATH="$PATH:/bin"  # for /bin/jupyter
```
One may already be there for other apps.

After install: Add themes

https://github.com/dunovank/jupyter-themes

Use within VSCode

Jupyter works great within VSCode.

See https://code.visualstudio.com/docs/datascience/jupyter-notebooks

Start Jupyter server
In a Terminal window, enter Jupyter (with & to run in background so you can keep working on the command line):
```
jupyter notebook --port 8888 notebook1.ipynb
```
A “&” is added to the end of the command above to run the server in background so that additional bash commands can execute (such as identifying the process number and cleaning up after shutdown).

The command above creates a (hidden) file .jupyter in your home folder, invokes a web server locally, and automatically pops up to your default browser a tab with url:

http://localhost:8888/tree

Alternately, to start without a web browser:
```
jupyter notebook --no-browser
```
The command response briefly shows a process number:
```
[1] 2809
```
If the command is run again on another Terminal, additional Notebook servers are invoked with the port number incremented from 8888.

Shutdown jupyter server

If the server was started in background, you can’t shutdown the server from Terminal by pressing control + C twice. You would need to run (such as):
```
kill -9 2809
```

Run Notebook UI

https://jupyter.readthedocs.io/en/latest/running.html

If Jupyter was started without specifying a Notebook file, you would see a tree listing.
Click a link to a .ipynb (Notebook) file listed, to see it’s code.
PROTIP: You can press the “Run” icon to run a single step. However, those familiar with the file can press the “»” button to run all cells (steps) in the file.

Each invocation opens a new browser window.

Keyboard Command Palette of shortcuts

NOTE: This does not work in Firefox and Internet Explorer, only in Chrome and Apple Safari.
When in command mode, click the little keyboard icon, called the “command palette”.

A panel appears with a search bar.
Press down arrow to scroll down.

Keyboard shortcuts are on the right side.
Enter a command to search for. Helpful for speeding up your workflow instead of looking around in the menus with your mouse.

See http://docutils.sourceforge.net/rst.html
Press Esc key.

Run All Cells
Click “Cell” in the menu and select “Run All”:

Kernel

Alternately, select the Kernel menu “Restart and Run All”:

DEFINITION: A “kernel” is each web page (window).
Create a kernel by clicking on the New drop-down and selecting one.
Click on tab File to show all the files and folders in the current directory.
Click Open.
List all the currently running notebooks by clicking on the Running tab.
Select “Python [Conda root]” for a new window (with tabs).

CAUTION: Just closing the browser leaves a kernel running.
To close and halt the kernel, select File, then Close and Halt.

Conda
cd to the folder containing the Jupyter notebook above.
A) Create a conda environment based on the requirements file:
```
conda install --file requirements.txt
```
B) Create a stand-alone environment named PDSH with Python 3.5 and all the required package versions:
```
conda create -n PDSH python=3.5 --file requirements.txt
```
Source activate the conda environment.
Install packages from within the Conda environment:

conda install jupyter notebook numpy matplotlib scikit-learn bokeh

Editing
Click on the Edit tab.

NOTE: Cells are where code is written and run.

Several of the selections have an icon equivalent.
Press Esc key.
Each cell can be changed by just clicking on it.

When the thick left border on a box is colored green, the box is in edit mode.

If you don’t see a blinking cursor in the cell, click in the cell.
To exit from edit mode to display mode, hold down Shift and press Enter.

Command mode

When a new cell is created, it is in command mode.

When the thick left border on a box is colored blue, the box is in command mode.

When the bar is blue, press H for a help screen about Keyboard shortcuts. It reads:

“Command mode binds the keyboard to notebook level actions.”

When the bar is blue, press A to create a new cell.

press B to create a cell below the currently selected cell.

Python Code
Click on the “Code” drop-down, which specifies what is typed in the cell (input box).
Select Markdown to format syntax for writing web content.
Type in some markdown text. See

https://daringfireball.net/projects/markdown/basics

https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

REMEMBER: Two dollar signs begin and end math entries.
To run the cell, click the >| icon (used to mean forward to the end) or press Shift + Enter or Control + Enter to run the Markdown cell.

Heading
Click the up/down arrows to position the cell above or below existing runned lines.
Select Heading and add a # to reduce the level before typing heading text.

Open and Rename Notebook
Click Files tab.
Click Open.

Select _____???
Click the checkbox to the right of the file name.
Click the Rename button.
Remove the “.txt” and press OK.
Click on the file name itself to open the file.

A new tab should appear.

Timings

To get the number of milliseconds a function took to run,
put %timeit in front of commands to invoke.

To get the number of seconds it took for a cell to run,
put %%timeit at the top of the cell.

Debugging

%pdb at the top of the cell turns debugging on.

https://docs.python.org/3/library/pdb.html

Visualization magic keywords

%matplotlib inline

See http://ipython.readthedocs.io/en/stable/interactive/magics.html for docs about magic commands.

nbextensions
Code to dynamically loaded:
```
%%javascript
IPython.load_extensions('custom_shortcuts');
```
See https://carreau.gitbooks.io/jupyter-book/content/Jsextensions.html

Raw NBConvert

NBConvert code modify the UI and behavior of Jupyter itself on the browser.

So Jupyter notebook extensions are written in javascript and CSS.

nbextensions are installed in the directory of the same name, either system wide or in your user profile. Their entry point is named extensionname.js

Output as HTML
To share a Notebook with others who do not have Notebook installed, convert the Notebook to HTML or Markdown.

PROTIP: Some prefer receiving Markdown text so they paste in blog editing software which formats the Markdown to their own liking.
```
jupyter nbconvert --to html notebook.ipynb
```
See https://nbconvert.readthedocs.io/en/latest/usage.html

See https://ipython.org/ipython-doc/1/interactive/nbconvert.html#notebook-format

GitHub

https://github.com/blog/1995-github-jupyter-notebooks-3 GitHub renders Jupyter Notebooks with Git Large File Support.

NBViewer

http://nbviewer.jupyter.org renders notebooks from a GitHub repo or from notebooks stored elsewhere.

Pandas Slideshows
Click the View tab.

See http://nbviewer.jupyter.org/format/slides/github/jorisvandenbossche/2015-PyDataParis/blob/master/pandas_introduction.ipynb#/

Turi (Dato) Python algorithms

GraphLab Create from Dato is no longer available since Dato was sold and an unrelated entity now uses their domain name.

It provided scalable “pre-implemented” ML algorithms using Python installed using Anaconda. Entire courses on its use is at:

https://www.coursera.org/learn/ml-foundations
https://www.turi.com/learn/userguide/
https://www.turi.com/products/create/docs/
https://github.com/learnml/machine-learning-specialization
https://www.coursera.org/learn/ml-clustering-and-retrieval/supplement/iF7Ji/software-tools-you-ll-need-for-this-course

When the one-year free license is over, note scikit-learn also uses Python with Anaconda.

https://www.youtube.com/watch?v=GxZUdZMPGpQ Large-Scale Deep Learning with TensorFlow Turi, Inc.

References

This is based on tutorials from several sources:

Jupyter Notebook Tutorial: Introduction, Setup, and Walkthrough

Additional

Clusters is not longer used to create multiple kernels used in parallel computing.

https://ipyparallel.readthedocs.io/en/latest/intro.html

Introduction to Data Science at UC Berkeley

Python for Data Analysis (Safari books online) by Wes McKinney

https://medium.com/analytics-vidhya/how-to-do-neural-style-transfer-in-python-bee3f0d1008b

More about Python

This is one of a series about Python:

bomonike

Google Colab (Colaboratory)

Photo Style Transfer

Style Transfer your picture’s URL

Sample Notebook files in Colab

Browser configuration on your laptop

Jupyter notebook and session files

Ignore checkpoints on push

macOS Server Installation

Server Installation

Manual install

After install: Add themes

Use within VSCode

Start Jupyter server

Shutdown jupyter server

Run Notebook UI

Keyboard Command Palette of shortcuts

Run All Cells

Kernel

Conda

Editing

Command mode

Python Code

Heading

Open and Rename Notebook

Timings

Debugging

Visualization magic keywords

nbextensions

Raw NBConvert

Output as HTML

GitHub

NBViewer

Pandas Slideshows

Turi (Dato) Python algorithms

References

Additional

More about Python