The Euclid Archive Processing and Data Distribution Systems:  A Distributed Infrastructure for Euclid and Associated Data

O. R. Williams; K. Begeman; D. Boxhoorn; B. Droge; T. A. Nutma; A. Tsyganov; E. A. Valentijn; W.-J. Vriend; C. Dabin

The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data

O. R. Williams^*, K. Begeman, D. Boxhoorn, B. Droge, T. A. Nutma, A. Tsyganov, E. A. Valentijn, W.-J. Vriend, C. Dabin

^*Corresponding author for this work

Astronomy

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

46 Downloads (Pure)

Abstract

The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST). The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and ten national Science Data Centres. The large data volumes demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX, iRODS, gridftp and Xrootd. No limitations are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be started at the most appropriate locations, or data transferred as necessary.

Original language	English
Title of host publication	Astronomical Data Analysis Software and Systems XXIX
Subtitle of host publication	Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX).
Editors	R. Pizzo, E.R. Deul, J-D. Mol, J. de Plaa, H. Verkouter
Publisher	Astronomical Society of the Pacific
Pages	291-294
Number of pages	4
Publication status	Published - 2020
Event	Astronomical Data Analysis Software and Systems XXIX: (ADASS XXIX) - Groningen, Netherlands Duration: 6-Oct-2019 → 10-Oct-2019

Publication series

Name	Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series
Publisher	ASP
Volume	527

Conference

Conference	Astronomical Data Analysis Software and Systems XXIX
Country/Territory	Netherlands
City	Groningen
Period	06/10/2019 → 10/10/2019

Access to Document

the proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX)Final publisher's version, 676 KBLicence: CC BY

http://adsabs.harvard.edu/abs/2020ASPC..527..291W

Handle.net

Persistent link

Cite this

Williams, O. R., Begeman, K., Boxhoorn, D., Droge, B., Nutma, T. A., Tsyganov, A., Valentijn, E. A., Vriend, W.-J., & Dabin, C. (2020). The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data. In R. Pizzo, E. R. Deul, J.-D. Mol, J. de Plaa, & H. Verkouter (Eds.), Astronomical Data Analysis Software and Systems XXIX: Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX). (pp. 291-294). (Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series; Vol. 527). Astronomical Society of the Pacific. http://adsabs.harvard.edu/abs/2020ASPC..527..291W

Williams, O. R. ; Begeman, K. ; Boxhoorn, D. et al. / The Euclid Archive Processing and Data Distribution Systems : A Distributed Infrastructure for Euclid and Associated Data. Astronomical Data Analysis Software and Systems XXIX: Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX).. editor / R. Pizzo ; E.R. Deul ; J-D. Mol ; J. de Plaa ; H. Verkouter. Astronomical Society of the Pacific, 2020. pp. 291-294 (Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series).

@inproceedings{4080d110747041d7b51211d2143e466c,

title = "The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data",

abstract = "The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST). The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and ten national Science Data Centres. The large data volumes demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX, iRODS, gridftp and Xrootd. No limitations are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be started at the most appropriate locations, or data transferred as necessary.",

author = "Williams, {O. R.} and K. Begeman and D. Boxhoorn and B. Droge and Nutma, {T. A.} and A. Tsyganov and Valentijn, {E. A.} and W.-J. Vriend and C. Dabin",

year = "2020",

language = "English",

series = "Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series",

publisher = "Astronomical Society of the Pacific",

pages = "291--294",

editor = "{ Pizzo}, {R. } and { Deul}, {E.R. } and { Mol}, J-D. and {de Plaa}, J. and Verkouter, {H. }",

booktitle = "Astronomical Data Analysis Software and Systems XXIX",

note = "Astronomical Data Analysis Software and Systems XXIX : (ADASS XXIX) ; Conference date: 06-10-2019 Through 10-10-2019",

}

Williams, OR, Begeman, K, Boxhoorn, D , Droge, B, Nutma, TA, Tsyganov, A , Valentijn, EA , Vriend, W-J & Dabin, C 2020, The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data. in R Pizzo, ER Deul, J-D Mol, J de Plaa & H Verkouter (eds), Astronomical Data Analysis Software and Systems XXIX: Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX).. Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series, vol. 527, Astronomical Society of the Pacific, pp. 291-294, Astronomical Data Analysis Software and Systems XXIX, Groningen, Netherlands, 06/10/2019. <http://adsabs.harvard.edu/abs/2020ASPC..527..291W>

The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data. / Williams, O. R.; Begeman, K.; Boxhoorn, D. et al.
Astronomical Data Analysis Software and Systems XXIX: Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX).. ed. / R. Pizzo; E.R. Deul; J-D. Mol; J. de Plaa; H. Verkouter. Astronomical Society of the Pacific, 2020. p. 291-294 (Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series; Vol. 527).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The Euclid Archive Processing and Data Distribution Systems

T2 - Astronomical Data Analysis Software and Systems XXIX

AU - Williams, O. R.

AU - Begeman, K.

AU - Boxhoorn, D.

AU - Droge, B.

AU - Nutma, T. A.

AU - Tsyganov, A.

AU - Valentijn, E. A.

AU - Vriend, W.-J.

AU - Dabin, C.

PY - 2020

Y1 - 2020

N2 - The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST). The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and ten national Science Data Centres. The large data volumes demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX, iRODS, gridftp and Xrootd. No limitations are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be started at the most appropriate locations, or data transferred as necessary.

AB - The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST). The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and ten national Science Data Centres. The large data volumes demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX, iRODS, gridftp and Xrootd. No limitations are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be started at the most appropriate locations, or data transferred as necessary.

M3 - Conference contribution

T3 - Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series

SP - 291

EP - 294

BT - Astronomical Data Analysis Software and Systems XXIX

A2 - Pizzo, R.

A2 - Deul, E.R.

A2 - Mol, J-D.

A2 - de Plaa, J.

A2 - Verkouter, H.

PB - Astronomical Society of the Pacific

Y2 - 6 October 2019 through 10 October 2019

ER -

Williams OR, Begeman K, Boxhoorn D , Droge B, Nutma TA, Tsyganov A et al. The Euclid Archive Processing and Data Distribution Systems: A Distributed Infrastructure for Euclid and Associated Data. In Pizzo R, Deul ER, Mol JD, de Plaa J, Verkouter H, editors, Astronomical Data Analysis Software and Systems XXIX: Proceedings of the 29th annual conference on Astronomical Data Analysis Software and Systems (ADASS XXIX).. Astronomical Society of the Pacific. 2020. p. 291-294. (Astronomical Data Analysis Software and Systems XXIX. ASP Conference Series).