open:wp3:jupyter-casa_repository
Differences
This shows you the differences between two versions of the page.
|
open:wp3:jupyter-casa_repository [2017/06/01 14:16] keimpema [Introduction] |
open:wp3:jupyter-casa_repository [2017/06/01 14:32] (current) keimpema |
||
|---|---|---|---|
| Line 6: | Line 6: | ||
| The size of astronomical datasets has increased dramatically over the years; terabyte sized datasets are no longer an exception. This trend will only accelerate; the SKA is expected to produce nearly 1 TB of archived data each day. This means that it will no longer be feasible for astronomers to download these huge datasets and perform the data reduction on their own machines, as is currently the practice. Instead the data reduction is likely to be done close to where the data is archived in central data processing centres, with the astronomer operating remotely on the data. | The size of astronomical datasets has increased dramatically over the years; terabyte sized datasets are no longer an exception. This trend will only accelerate; the SKA is expected to produce nearly 1 TB of archived data each day. This means that it will no longer be feasible for astronomers to download these huge datasets and perform the data reduction on their own machines, as is currently the practice. Instead the data reduction is likely to be done close to where the data is archived in central data processing centres, with the astronomer operating remotely on the data. | ||
| - | One way of facilitating this is through [[http://jupyter.org/|Jupyter]] notebooks . Jupyter is a web-based application which allows users to create interactive notebooks which can include annotated text and graphics as well as executable code. Currently Jupyter supports more than 40 different programming languages, including Python, R, and Matlab. Jupyter is designed be extended and makes it easy to add additional languages. | + | One way of facilitating this is through [[http://jupyter.org/|Jupyter]] notebooks. Jupyter is a web-based application which allows users to create interactive notebooks which can include annotated text and graphics as well as executable code. Currently Jupyter supports more than 40 different programming languages, including Python, R, and Matlab. Jupyter is designed be extended and makes it easy to add additional languages. |
| - | As part of the Obelics work-package we have created a Jupyter kernel for CASA, a widely-used software package for processing astronomical data. The kernel allows all CASA tasks to be run from inside a Jupyter notebook, albeit non-interactively. Tasks which normally spawn a GUI window are wrapped so that their output is saved to an image instead, which is then displayed inside the notebook. | + | As part of the Obelics work-package we have created a Jupyter kernel for [[https://casa.nrao.edu/|CASA]], a widely-used software package for processing astronomical data. The kernel allows all CASA tasks to be run from inside a Jupyter notebook, albeit non-interactively. Tasks which normally spawn a GUI window are wrapped so that their output is saved to an image instead, which is then displayed inside the notebook. |
| The notebook format also has the great advantage that all steps of the data reduction are preserved inside the notebook. This means that the whole data reduction process is self-documenting and fully repeatable. It also allows users to very easily make changes to their pipeline and then rerun the pipeline steps affected. | The notebook format also has the great advantage that all steps of the data reduction are preserved inside the notebook. This means that the whole data reduction process is self-documenting and fully repeatable. It also allows users to very easily make changes to their pipeline and then rerun the pipeline steps affected. | ||
| Line 17: | Line 17: | ||
| Installation is a simple as executing: ''docker pull penngwyn/jupytercasa'' | Installation is a simple as executing: ''docker pull penngwyn/jupytercasa'' | ||
| + | |||
| + | The source code is located here: https://github.com/aardk/jupyter-casa | ||
| ===== Usage ===== | ===== Usage ===== | ||
| - | Even though we wrap all CASA tasks so that they will not launch a GUI window, the QT based CASA tasks still require X11, unfortunately. Tasks such as plotms won't start unless X11 is working even when it doesn't even open a window. Therefore the local X11 socket needs to be shared with Docker container. | + | Even though we wrap all CASA tasks so that they will not launch a GUI window, the QT based CASA tasks still require X11, unfortunately. Tasks such as //plotms// won't start unless X11 is working even when it doesn't even open a window. Therefore the local X11 socket needs to be shared with Docker container. |
| The simplest incantation to start JUPYTER on a recent Ubuntu: | The simplest incantation to start JUPYTER on a recent Ubuntu: | ||
| Line 28: | Line 30: | ||
| '' | '' | ||
| - | Note that the '--rm' option will make DOCKER delete the container after use. | + | Note that the ''--rm'' option will make DOCKER delete the container after use. |
| - | Of course the above example is not very usefull as the container will not be able to access locally stored measurement sets. To add a data directory to the DOCKER container is, fortunately, very simple using the -v option: | + | Of course the above example is not very useful as the container will not be able to access locally stored measurement sets. To add a data directory to the DOCKER container is, fortunately, very simple using the ''-v'' option: |
| '' | '' | ||
| Line 36: | Line 38: | ||
| '' | '' | ||
| - | Where PATH_TO_DATA_DIR should be replaced with the full path to your local data directory. | + | Where ''PATH_TO_DATA_DIR'' should be replaced with the full path to your local data directory. |
| - | + | ||
| - | The above examples use a JUPYTER kernel which is baked into the DOCKER image. It is also possible to use the GITHUB development version within the CASA container, from the root of the source tree run: | + | |
| - | + | ||
| - | '' | + | |
| - | docker run --rm -p 8888:8888 -i -t -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $PWD/jupyter:/home/jupyter/.local/share/jupyter -v $PWD/python/casapy:/home/jupyter/.local/lib/python2.7/site-packages/casapy -v PATH_TO_DATA_DIR:/home/jupyter/data penngwyn/jupytercasa /bin/sh -c "jupyter notebook" | + | |
| - | '' | + | |
| ===== Examples ===== | ===== Examples ===== | ||
| - | In the examples directory there is a notebook which contains the NRAO continuum VLA tutorial. To run that code locally be sure to download the data files from the NRAO wiki. | + | We have created an example [[https://github.com/aardk/jupyter-casa/blob/master/examples/vla-cont-tutorial.ipynb|notebook]] which contains the NRAO continuum VLA tutorial. To run the notebook locally download the data files from the [[https://casaguides.nrao.edu/index.php/VLA_Continuum_Tutorial_3C391-CASA4.7|NRAO wiki]] and make the data available to the DOCKER container as is explained above. |
| - | + | ||
| - | Also don't forget to make the directory available to the DOCKER container using the -v option as is explained above. | + | |
| - | + | ||
open/wp3/jupyter-casa_repository.1496319397.txt.gz · Last modified: 2017/06/01 14:16 by keimpema
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
