3D waterfall plot in WebGL

In 2016, I finally learned OpenGL.

So I decided to ditch THREE.js and rewrite my old Waterfall-Plot project in pure WebGL. The only external Javascript library used is gl-matrix for matrix transformations.

All the line offsetting (z-direction in time, y-direction for frequency magnitude) is now done directly in the shader. This makes the animation much smoother, as no THREE.js points/lines have to be created and destroyed anymore. It also allows more flexibility in color effects.

To hide the lines behind each other, a gl.LINE_STRIP of triangles is rendered for each line. DEPTH_TEST is not enabled, the lines and triangles are just rendered in the right order.

Live demo

Press zxcvb for some color presets.

Use q for enabling/disabling lines, and wertyu for changing the line color. Use a for enabling/disabling the stripes, and sdfghj for changing their color. Use 1234 to change line thickness.

Sample 1 Sample 2
Sample 3 Sample 4
Some samples.

Find the project on Github.

This website has been moved!

This website used to be hosted on https://people.ee.ethz.ch/~muejonat/.

My studies at ETH will end soon and I will consequently lose the home directory web hosting there. To not lose the audience (and Google ranking), I decided to act early enough (1 year) and redirect everything to the new address https://jo-m.ch/ via 301.

Redirect without mod_rewrite

Because

  • mod_rewrite via .htaccess does not seem to work on people.ee.ethz.ch
  • or I did not manage to get it to work
  • and was too lazy to ask support

my solution looks like this:

.htaccess

ErrorDocument 404 /~muejonat/index.php

index.php

<?php
    preg_match('#/~muejonat/(.*)#', $_SERVER[REQUEST_URI], $matches);
    header('Location: '.'https://jo-m.ch/'.$matches[1], true, 301);
?>

Farewell, http://people.ee.ethz.ch/!

ETH supercomputing for beginners

Disclaimer: for this to work, you must be an ETH Zurich affiliated person and own a nethz-account.

So, you wanna play around a bit with machine learning. Or run a crazy particle physics simulation. But you only have a lame Macbook. Your gaming rig would do the job, but its fans are spinning so loud your neighbors complained. Solution? Just use ETHs Euler general purpose super computer! Even though they explicitely write that Euler is “not a supercomputer”, supercomputer just sounds cooler than “general purpose” computer.

So called shareholders (lab groups and ETH departments) invested money to own a reserved percentage of Eulers computing power. However, there is also a slice reserved for us students. The best thing about it? It works without need for any bureaucracy. Just log in and start.

Python example

We will run an example from the scikit-learn website for demonstration. There will be some some small changes, because we have no GUI. But first, let’s log in. Important: you can only log in to Euler from within the ETH network or when connected via VPN.

ssh <your nethz-name>@euler.ethz.ch

You will be greeted with a disclaimer you have to accept by typing Yes the first time. Then, we can start by loading the python module.

module load python/2.7
# get sample script from scikit-learn
wget http://scikit-learn.org/stable/_downloads/plot_image_denoising.py

Now you’ll have to modify the script to make it run without X11. For this, we have to tell matplotlib to write to disk instead trying to display the images directly. Modify the downloaded file at the top and the bottom to look like this:

# at the top of the file,
# after the long introductionary comment
print(__doc__)

from time import time

# those two lines must be inserted here
import matplotlib
matplotlib.use('Agg')

import matplotlib.pyplot as plt
import numpy as np

...

# bottom
# replaced plt.show() by the following line
plt.savefig('plot.png')

When you submit a job for the batch processing system, it will inherit the current environment. As we already loaded the python module, we are now ready to go:

# submit job
bsub "python plot_image_denoising.py"
> Generic job.
> Job <9613719> is submitted to queue <normal.4h>.

# you can now check the status of your job via bjobs
bjobs
> JOBID      USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
> 9613719    muejo   RUN   normal.4h  euler01     e1096       *oising.py Aug 25 00:34

After some time, in the directory a file called lsf.o<jobid> will appear, besides the plot.png our script generated. That’s it, we’re done!

Remarks:

  • Your job will be killed after 4 hours. You can use an option -W hh:mm with bsub let it run longer, but it will wait longer in the queue.
  • Same is valid for CPU cores (-n 2 uses 2 cores instead of 1) and memory (-R "rusage[mem=2048]" uses 2GB per core). You can use up to 48 cores at the same time (check using busers, MAX). If you submit several jobs, requiring more than 48 jobs in total, they will be run sequentially. If you submit a single job requiring more than 48 cores, it will probably be stuck in the queue forever.

Further information (only accessible from within ETH network):

If you need some special packages

You can install libraries via PIP locally in your home dir.

mkdir -p $HOME/python/lib64/python2.7/site-packages
export PYTHONPATH=$HOME/python/lib64/python2.7/site-packages:$PYTHONPATH
module load python/2.7

# now, install e.g. theano
python -m pip install --install-option="--prefix=$HOME/python" theano

Some packages require Euler module dependencies. For example, if you wanna use h5py, you have to load the hdf5 module before loading the python module:

module load hdf5
module load python/2.7.2
python
>>> import h5py
# works!

GPU

Only the not publicly-accessible Leonhard cluster has GPU nodes. So if you need that, you will have to ask Cluster Support.

Support

I am a student who does not know anything about scientific computing, let alone ETHs HPC infrasctructure. This blog post just tries to provide some guidance for students. I do not provide any support. Also, I am not responsible if the admins get angry at you because you ran your stuff on a login node.

Find support here

For D-ITET students

There is an additional resource for D-ITET students. You can run jobs on the idling tardis machines (in the ETZ computer rooms). If you ever wondered, that is the reason for the “do not turn off” labels. The batch system is very similar to the one on Euler. Find more infos in the D-ITET Computing Wiki.

Updates

This post was updated on Jan 26 2017 to reflect the decomissioning of the old Brutus cluster and its Wiki. The new Wiki is now to be found here (Only from within ETH network!).

Javascript 3D waterfall plot

For a long time, I wanted to try out WebGL. After some initial research, it became clear that it is not that simple to use WebGL without having prior experience with OpenGL. Thats why I settled with THREE.js. This library abstracts away all the WebGL things and gives you a nice 3D interface.

I have for a long time liked Joy Divisions Unknown Pleasures album cover. You can find something about its history here.

The idea now was to recreate a live version of this cover, using audio data from the PC microphone. So I also could learn the basics of webRTC!

Live demo (click here)

The result.

Find the project on Github.

Wifi monitor on a Raspberry Pi

I had a Raspberry Pi and an old analog volt meter lying around. To finally do something “useful” with it, it was decided that our CoWorking Space needs a WiFi detector.

Functionality in short: It captures WiFi packets from the air, and counts them. This then gets mapped to a scale from 0 to 10, and displayed on a rainbow-colored LED chart.

Hardware

This runs on a Raspberry Pi Model B, although all the other models should work too.

I am using a $8 WiFi dongle from Dealextreme. It just works™ after plugging into the RasPi. lsusb reports it as Ralink Technology, Corp. RT5370 Wireless Adapter with ID 148f:5370.

Be sure to have an ethernet connection to your Raspberry Pi! During setup we will change the WiFi device to monitor mode and thus lose the wifi connection. So you could lock out yourself if you SSH into it using WiFi (although a restart will fix it).

System setup

sudo apt-get -y update
sudo apt-get -y install iw python-scapy tcpdumppython-dev
sudo pip install rpio

# add a monitor device

# test if monitoring works
sudo iw phy phy0 interface add mon0 type monitor
sudo iw dev wlan0 del
sudo ifconfig mon0 up
sudo iw dev mon0 set channel 6

# tcpdump should continuously list captured packages. Kill it with Ctrl-C.
sudo tcpdump -i mon0 -n

# teardown, back to normal wifi operation
sudo iw dev mon0 del
sudo iw phy phy0 interface add wlan0 type managed

Install our script

scp -r raspi-mon pi@<ip_of_your_pi>

On your Pi, add the following line to /etc/rc.local (to enable autostart of the script):

sudo python /home/pi/raspi-mon/server.py &

LEDs

Now, you have to wire up some LEDs from the Raspberry Pi GPIO pins. The RasPi has no problem outputting 10x 20mA, so you can connect them directly. Don’t forget to add a resistor to each LED although, except you want to destroy them in short time. I used ports 2, 3, 4, 17, 27, 22, 10, 9, 11, 7. If you use different ports, you should change this line.

I used 2 LEDs per color, red, orange, green, blue, white.

RasPi pin wiring LEDs
LEDs and pins on the RasPi.

Software explanation

Used software:

  • Python
  • ScaPy for capturing packets
  • iw for managing the WiFi device
  • SQLite for data storage and simple analytics

I decided to use SQLite. It serves for multiple things simultaneously:

  • Thread-safe communication between LED and capturing thread
  • Easy data analytics (just SQL)
  • Cheap storage on disk (well… SD-card)

There are some things running in parallel.

  • One thread runs Scapy and sniffs. It logs all the beacon and data packets to the database.
  • Another thread (ChannelHopper) changes the wifi channel every 0.3 seconds. This is because we want to listen on all available channels, not just one..
  • The third and last thread (main thread) pulls the data from the SQLite database and maps the captured activity to the LEDs. It then sleeps for one second, and repeats.

Conclusion

It works, although only on the 2.4GHZ band (the cheap adapter does not provide 5GHZ access). When I download something on my phone using a 2.4GHZ channel, I can see the LED bar going up! The whole thing is deployed in a cardboard box. It just runs after powering on (plugging in) the RasPi.

It works!