The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data

O. R. Williams*, K. Begeman, D. Boxhoorn, B. Droge, T. A. Nutma, A. Tsyganov, E. A. Valentijn, W.-J. Vriend, C. Dabin

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

7 Downloads (Pure)

Abstract

The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST). The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and ten national Science Data Centres. The large data volumes demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX, iRODS, gridftp and Xrootd. No limitations are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be started at the most appropriate locations, or data transferred as necessary.
Original languageEnglish
Title of host publicationAstronomical Data Analysis Software and Systems XXIX
Subtitle of host publication Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX).
EditorsR. Pizzo, E.R. Deul, J-D. Mol, J. de Plaa, H. Verkouter
PublisherAstronomical Society of the Pacific
Pages291-294
Number of pages4
Publication statusPublished - 2020
EventAstronomical Data Analysis Software and Systems XXIX: (ADASS XXIX) - Groningen, Netherlands
Duration: 6-Oct-201910-Oct-2019

Publication series

NameAstronomical Data Analysis Software and Systems XXIX. ASP Conference Series
PublisherASP
Volume527

Conference

ConferenceAstronomical Data Analysis Software and Systems XXIX
Country/TerritoryNetherlands
CityGroningen
Period06/10/201910/10/2019

Cite this