The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data

O. R. Williams*, K. Begeman, D. Boxhoorn, B. Droge, T. A. Nutma, A. Tsyganov, E. A. Valentijn, W.-J. Vriend, C. Dabin

*Bijbehorende auteur voor dit werk

OnderzoeksoutputAcademicpeer review

15 Downloads (Pure)

Samenvatting

The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST). The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and ten national Science Data Centres. The large data volumes demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX, iRODS, gridftp and Xrootd. No limitations are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be started at the most appropriate locations, or data transferred as necessary.
Originele taal-2English
TitelAstronomical Data Analysis Software and Systems XXIX
Subtitel Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX).
RedacteurenR. Pizzo, E.R. Deul, J-D. Mol, J. de Plaa, H. Verkouter
UitgeverijAstronomical Society of the Pacific
Pagina's291-294
Aantal pagina's4
StatusPublished - 2020
EvenementAstronomical Data Analysis Software and Systems XXIX: (ADASS XXIX) - Groningen, Netherlands
Duur: 6-okt.-201910-okt.-2019

Publicatie series

NaamAstronomical Data Analysis Software and Systems XXIX. ASP Conference Series
UitgeverijASP
Volume527

Conference

ConferenceAstronomical Data Analysis Software and Systems XXIX
Land/RegioNetherlands
StadGroningen
Periode06/10/201910/10/2019

Citeer dit