open:wp4:wp4techforum2
Table of Contents
ASTERICS DADI Technology Forum 2
The Forums enable technology discussion involving the framework developers and ESFRI project staff, to develop technical collaboration and exchange good practice - establishing a European strategy for IVOA and RDA activities
(Top level DADI page: WP4: Data Access, Discovery and Interoperability (DADI))
Dates
Meeting: 7 & 8 March, 2016
Location
The meeting will be held on the Pollock Halls campus in the Pollock Room in St Leonard's Hall
Hotels
Please book your own accommodation.
To book campus accommodation, visit the booking page, and quote promotion code: CONF16. This will provide 15% off accommodation at Masson House and Salisbury Green Hotel at Pollock Halls.
There are many other Hotels and Guest Houses in the vicinity of Pollock Halls. More information can be found here and here
Dinner
A working dinner will be held Monday evening. This needs to be confirmed as soon as possible so please indicate whether you wish to attend.
Agenda
| Time | Title | Speaker | Materials |
| Day 1 | Monday, March 7th | ||
| 10:00 | Coffee | ||
| 10:15 | Welcome and round table intros | Keith / Mark A | |
| 10:40 | DADI status | Françoise | DADI Status |
| 11:00 | EM follow-up of gravitational wave candidates | Giuseppe | Gravitational Waves |
| 11:20 | HiPS - Hierarchical Progressive Survey | Pierre | HiPS |
| 11:40 | A&A systems developed for CTA | Mathieu | A&A for CTA |
| 12:00 | Applying A&A to server/client VO scenario | Marco | A&A Server-client |
| 12:20 | Applicable conditions for A&A (web, non web resources, etc.) | Marco | A&A applicable conditions |
| 12:40 | VODML from a VODSL perspective | Paul | VODSL |
| 13:00 | Lunch | ||
| 14:00 | Links between Provenance Data Model prototypeand the IVOA Dataset Metadata data Model | Mireille | Provenance |
| 14:15 | DAL protocols for ESFRI projects / current status | Francois / Chaitra | DAL Protocols |
| - IVOA DAL context status | |||
| - current development at CDS : DataLink integration within Aladin | |||
| 14:50 | Aladin Lite and usage of the MOCServer | Thomas | MOCServer |
| 15:10 | SAMP over HTTPS | Mark T | tlsamp |
| 15:30 | Afternoon tea | ||
| 15:50 | Intro to Hack-a-thon | Keith | |
| 16:00 | Hack-a-thon | ||
| 17:30 | Close | ||
| 19h | Tech Forum Dinner To be held in the Nelson Room, St Leonard's Hall | ||
| Day 2 | Tuesday, March 8th | ||
| 09:30 | Description and discovery of VizieR resources in VO registries) | Sébastien | VizieR resource discovery |
| 09:50 | Registry | Markus | Registry Notes |
| 10:50 | Coffee | ||
| 11:10 | SODA – what's all the fuss about on the DAL list? | Markus | SODA Notes |
| 11:30 | Madrid School | Françoise | School2015 |
| 11:50 | Engaging with major astronomy projects at the next IVOA meeting | Mark A | Engagement |
| 12:10 | Discussion : Engagement & IVOA Interop prep | ||
| 13:00 | Lunch | ||
| 14:00 | Hack-a-thon | ||
| 15:30 | Afternoon tea | ||
| 15:50 | Hack-a-thon | ||
| 17:00 | Conclusions | Keith / Mark A | |
| 17:30 | Close | ||
Breakout sessions
Monday
| 1. Provenance data model (Notes author : Mireille) |
| minutes by Mireille Louys draft to be checked with group. |
| Francois Bonnarel, Markus Nullmeier, Mireille Louys, Matthieu Servillat, Julien LeFaucheur, Chaitra ( CDS contract) |
| (i). Which existing generic tools could we re-use? Prov-N used in the RAVE use-case Prov Store (Southampton) offers a software suite to handle the entity/activity/agent pattern. It is based on the Prov package in Python that can be used directly. Will be explored further by Luth ( Mathieu, Julien ) |
| (ii). Format for serialising the metadata JSON prov-XML TAP-Schema Fits- keywords??? extra header Need for a bundle to wrap all together: data (fits) +Instrument files+ IVOA metadata |
| (iii). Modeling - which part to re-use from Obscore - how to bind IRF , instrument config, etc. - need for a version tag in the Method metadata |
| (iv). Usecases to clarify need for practical use-cases for all projects: *CTA Julien & Mathieu& Catherine ? a possible use-case: spectrum reconstruction as UWS service in LUTH. |
| (v). RAVE / IAP check with Kristin |
| Pollux Michèle |
| SVOM starting the effort(Laurent michel , XMM-High energy physics team, Strasbourg Observatory) |
| (vi). Questions to discuss Mireille: how could we disentangle the different layers: - science use-case: what the scientist needs to see , and provide as input data and parameters - chain of algorithms : description of methods and params. - workflow execution : job stalled, job terminated, files in , files out , config , etc. |
| (vii). Proposed meetings Wish to have an IVOA tech meeting together : Paris? Strasbourg? Heidelberg? difficult to plan before IVOA :may be in June-July? |
| (viii). IVOA DM session ask for a provenance DM session at next interop in Cape town: think about targeting some didactic presentation for the audience : many newcomers to the VO are expected. |
| 2. HiPS / Registry discussion (Notes author : Pierre) |
| 4 participants : Thomas Boch, Sebastien Derrière, Pierre Fernique & Markus Deimleitner |
| A good debate, with a lot of exchanges, lots of semantic explorations (creator vs publisher, resource vs service, …) |
| We did a beautiful loop to come back to the agreement that we had at the beginning of the discussion (publisher on App mailing list and presented in the Pierre F's HiPS talk) |
| 3. GWsky discussion (Notes author : Giuseppe) |
| Interesting discussions on the possibility of implementing MOC - dedicated to the coverage manipulations - and HiPS to manipulate data (pixels, catalog, sources, etc.). This will be done using the Aladin script commands via SAMP. In particular, the HEALPix pixels contained in a certain probability will be considered for faster loading of catalogs or images. |
In addition, we have discussed to develop a graphical user interface (Tkinter?) and to redirect the output of GWsky to a printable page. In general, the use of SAMP has been highly appreciated and encouraged for any future development
| 4. ADQL (Notes author : Dave) |
| Participants : Dave Moris & Marco Molinaro |
| Planning meeting concerning next steps for the ADQL specification. |
| OFFSET/LIMIT The current consensus on the mailing list is to accept OFFSET without LIMIT. TODO - Dave : Write a page on the IVOA wiki summarising the OFFSET/LIMIT syntax for each of the main platforms, PostgreSQL, MySQL, SQLServer, Oracle etc. If a database platform requires LIMIT, the proposed work around is to specify either (MAX_ROWS + 1) or (MAX_LONG). However, many of the text books and online manuals describe using OFFSET with LIMIT. TODO - Dave : Propose to the mailing list that we accept OFFSET with LIMIT, allowing LIMIT as a synonym for TOP. The only reason for this is to make ADQL compatible with 90% of the text books and online manuals. Hopefully this should be a short yes/no discussion and then accept the consensus for the v2.1 specification. TODO - Dave : Update the text of the v2.1 specification to include OFFSET (with/without LIMIT as agreed) |
| Hexadecimal literals Adding hexadecimal literals to the BNF grammar has raised some issues. The simple way to add them would be to extend numeric literal to allow both decimal and hexadecimal values. If we follow a strict interpretation of the BNF, then allowing both decimal and hexadecimal values for numeric literals causes some odd side effects, particularly with scientific notation and exponents. TODO - Dave : Write a page on the IVOA wiki detailing the issues. TODO - Dave : Propose to the mailing list that we accept a) Numeric literals may contain both decimal and hexadecimal values but there are places where hexadecimal values do not make sense, and developers should do the right thing. and b) The BNF grammar is not a definitive machine readable specification and there may be some unintended side effects in it. TODO - Dave : If consensus is to accept a flexible interpretation of the BNF, update the text of the v2.1 specification to reflect this. If consensus is for a strict interpretation of the BNF, then discussion on validating a strict BNF grammar will need to be continued at the May interop. |
| Crossmatch function The mailing list has yet to agree on a consensus for the crossmatch function. TODO - Marco : Write an email to the mailing list summarising the options and starting a new discussion thread. TODO - Dave : If consensus is reached, update the text of the v2.1 specification to include the crossmatch function. If no consensus is reached, then discussion of the crossmatch function can be continued at the May interop. |
| Specification review The plan is to try to resolve the outstanding issues during March, and have an updated version of the specification ready for review in April. If we are not able to reach consensus on an issue, then it should be deferred to the next version of the specification. |
| 5. Rendering plots server-side (Notes author : Mark) |
| Discuss options for server-side rendering of plots using topcat/plot2 infrastructure and a web browser with small(?) amounts of javascript Looks like it should be possible to make a start on it without too much difficulty. Mark to have a first go, Stelios may find time for some work on it in future if javascript expertise is required Future progress depends on our other commitments. |
| Discuss some technical issues related to auto-completion of ADQL queries. |
Tuesday
| 6. A&A (Notes author : Marco) |
| Participants: Giuliano Taffoni (skype) Mark Taylor Stelios Voutsinas Pierre Fernique Paul Harrison Keith Noddle Dave Morris Mathieu Servillat Thomas Boch |
| IVOA SSO-2.0 and CDP-1.0 standards are what is available in the VO to secure resources/services and delegate user certification. This standards seem to cover the needs on the server side (maybe CDP needs a refurbish due to technology changes) but the user authentication and authorization from the client side point of view is an open question. |
| Currently everything has worked against public services, and HTTPBasic authentication proved to be enough when data policies where of concern from data providers. |
| The participants discussed about whether it would be needed to introduce more complex solutions for authentication at present time and what the uses cases for this were. |
| The outcome was that: use cases probably are variations on: - a user that authenticates and then tries to access to different services: how does the identification works in this case - a user consumes a secured server and tries to push the results to a different one (again secured) |
| In a sense there are ongoing projects that try to solve this, in terms of provider federation, but not from the client side. |
| It seems to be no urgent matter to solve, nonetheless thinking of solutions to these use cases in advance can be useful to be prepared when the issue will became a critical one, and may also be useful if some ESFRI partners (instead of the full VO community) decide to pursue authentication when providing data and services (some of them are already developing AuthN&AuthZ infrastructures with identity providers of their own, e.g. CTA and SKA). |
| A solution for the client side, i.e. for applications that run on the user's desktop, could be, e.g., having a local authentication tool to act as a proxy to handle the SSL and authentication delegation towards the accessed resources. This would act as a sort of secondary HUB to the SAMP one, letting applications work mainly as they do now, with this proxy taking care of passing on the credential tokens or certificates to the requesting resources. |
| It could also be a way to solve the SAMP issue with the web profile and mixed HTTP/HTTPS connections. |
| Oncoming Northern Spring Interop (Cape Town, May 2016) will probably be a good place to continue on this discussion. |
| 7. pgSphere (Notes author : Markus N.) |
| minutes by Markus Nullmeier |
| Dave Morris, Keith Noddle, Markus Nullmeier, Mathieu Servillat, Mike Sipior (via Skype), Paul Harrison, Stefano Zampieri, Stelios Voutsinas |
| (i). An update on the state of pgSphere - Markus joined forces with the developers from Moscow, his work on the code was merged by Alexander Korotkov into the master branch of https://github.com/akorotkov/pgsphere, which is the official pgSphere repository for the time being. - A new official release of pgSphere is imminent, hopefully in a few weeks at latest. This release will support PostgreSQL releases 9.1 and later (i.e., all supported ones). It also uses the 9.1+ “CREATE EXTENSION pg_sphere;” SQL syntax to activate pgSphere for a database. - There will be a new home page for the project soon, together with a repository and a mailing list for both users and developers (everyone present accepted the invitation to become a member of this list). Then, a pointer to the new website will be put on the old project page on PgFoundry. - The Russian developers are still active in further improving the already much faster indexing code (one order of magnitude in the present code base) that was released on Github in 2014 (A. Korotkov: "A new double sorting-based node splitting algorithm for R-tree", Programming and Computer Software 38(3), 109-118). - The pgSphere documentation explains how to use its spatial indexing, which uses 3d R-trees on the unit sphere. - If somebody should require commercial support for pgSphere, Postgres Professional (info@postgrespro.ru) would certainly be up to the task. - Extending pgSphere to provide MOC as an additional data type for objects on the surface of the unit sphere (such as circles and polygons that were in pgSphere from the start) was delayed by several other things, but now it has very high priority for Markus. There already exists a concept to do proper (“high-order”) indexing support to accelerate SELECT operations on MOCs to be faster than sequential scan (just by viewing a MOC as a set of numbers), but this requires PostgreSQL core support (written by Alexander Korotkov), which is hopefully going to be accepted for inclusion into PostgreSQL 9.6 (due to be released in 2016). |
| (ii). Installing pgSphere - compiling the latest official release 1.1.1 from 2010 works only up to PostgreSQL release 9.1, this has created serious problems for at least one of the attendants (a patched version for later PostgreSQL releases is available as https://github.com/mnullmei/pgsphere/archive/fixes-1-1-1.tar.gz , users of Debian 8 (Jessie) may install a packaged version of this as “postgresql-9.4-pgsphere” from http://soft.g-vo.org/repo – additional information provided after the session) - Users of Redhat-based Linux distributions may install pgSphere 1.1.1 for a PostgreSQL from the PostgreSQL Yum Repository as the package “pgsphere95” for PostgreSQL 9.5 / etc. - Packaging pgSphere for the Debian Linux distribution (which will proliferate into Ubuntu), and also for the PostgreSQL Apt Repository (analogous to the PostgreSQL Yum Repository) is planned for the time after the upcoming pgSphere release - for pgSphere on Docker see further below. |
| (iii). pgSphere for DaCHS users - DaCHS support for the upcoming pgSphere release is under way. - DaCHS users should not upgrade to pgSphere from https://github.com/akorotkov/pgsphere just yet. |
| (iv). Use cases - image services, a feature that most other spatial database extensions lack - using the Java taplib libraries for TAP and ADQL to re-use existing Java code - (potential) replacement of non-PostgreSQL data base systems |
| (v). pgSphere on Docker - Dave gave an demonstration of effortlessly running different PostgreSQL releases under Docker. - Availability of pgSphere for Docker would be of considerable help to several people. - There is also some activity to provide DaCHS images for Docker. |
| (vi). Science-based common test cases - Dave instigated the idea to jointly create a set of common science-based test cases for ADQL, cone search, etc., with known results in VOTable files. - These would certainly also help pgSphere. - Markus mentioned http://skyview.gsfc.nasa.gov/xaminblog/index.php/tag/pgsphere – it is unknown if the bugs described there would still show up with the present codebase of pgSphere. |
| 8. SODA(Notes author : François) |
| SODA is the last “open” element of the “multi-dimensional standard caravan”, a top priority of the IVOA as well as ASTERICS. The debate on architecture for expanding the requirements beyond those defined by the IVOA Committee for Scientific Priorities continued during the Hack-A-Thon and it was not possible to reach a consensus at that time. |
| NB (added after the Cape Town Interop): The discussion continued in the IVOA DAL Working Group after the meeting and a consensus was finally reached on an incremental approach. It was presented in Cape Town and is the basis of SODA Proposed Recommendation. |
Organisation
Program Organising Committee:Keith Noddle, Mark Allen (co-chairs)
Local Organising Committee: Keith Noddle, Nathalie Dupin, Andrew Lawrence
Contact: ktn (at) roe.ac.uk
Participants
| Name | Affiliation | Menu |
| Andrew Lawrence | IfA, Edinburgh | |
| Bob Mann (excused) | IfA, Edinburgh | n/a |
| Carlos Rodrigo | INTA | |
| Chaitra Suvarna | CDS, Observatoire de Strasbourg | |
| Clive Davenhall | IfA, Edinburgh | n/a |
| Dave Morris | IfA, Edinburgh | |
| François Bonnarel | CDS, Observatoire de Strasbourg | |
| Francoise Genova | CDS, Observatoire de Strasbourg | |
| Giuliano Taffoni | INAF - OATs | |
| Giuseppe Greco | Università degli Studi di Urbino and INFN-Firenze | n/a |
| Julien Lefaucheur | LUTH / Observatoire de Paris-Meudon | |
| Keith Noddle | IfA, Edinburgh | V |
| Marco Molinaro | INAF - OATs | |
| Mark Allen | CDS, Observatoire de Strasbourg | |
| Mark Holliman | IfA, Edinburgh | n/a |
| Mark Taylor | University of Bristol | V |
| Markus Demleitner | ARI, UHEI, Heidelberg | V |
| Markus Nullmeier | ARI, UHEI, Heidelberg | V |
| Mathieu Servillat | LUTH / Observatoire de Paris-Meudon | |
| Mike Sipior (excused) | ASTRON/LOFAR | |
| Mireille Louys | CDS, Observatoire de Strasbourg | |
| Paul Harrison | University of Manchester | |
| Pierre Fernique | CDS, Observatoire de Strasbourg | |
| Rob van der Meer | ASTRON/LOFAR | |
| Sebastien Derriere | CDS, Observatoire de Strasbourg | |
| Stefano Zampieri | ||
| Stelios Voutsinas | IfA, Edinburgh | |
| Thomas Boch | CDS, Observatoire de Strasbourg | |
28|24 participants
Menu
- Hot Scottish smoked salmon, lemon, capers & black pepper
- Supreme of chicken stuffed with haggis in thyme scented whisky cream sauce
- Drambuie & stem ginger cranachan in a brandy snap basket
- Filter coffee & petits fours
Vegetarian Starter
- Sweet & sour vegetable spring roll with a fresh mango salsa
Vegetarian Main
- Sweet potato, squash & coriander tart with crispy sweet pickled vegetable salad
open/wp4/wp4techforum2.txt · Last modified: 2016/12/20 09:11 by genova
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
