The Role of the Euclid Archive System in the Processing of Euclid and External Data

O. R. Williams; K. Begeman; D. Boxhoorn; B. Droge; A. Tsyganov; J. P. McFarland; E. A. Valentijn; W. J. Vriend; C. Dabin

The Role of the Euclid Archive System in the Processing of Euclid and External Data

O. R. Williams, K. Begeman, D. Boxhoorn, B. Droge, A. Tsyganov, J. P. McFarland, E. A. Valentijn, W. J. Vriend, C. Dabin

Astronomy

Research output: Chapter in Book/Report/Conference proceeding › Chapter › Academic › peer-review

90 Downloads (Pure)

Abstract

Euclid is an ESA M2 mission which will create a 15,000 square degrees space-based survey: the Euclid Archive System (EAS) is a core element of the Science Ground Segment (SGS) of Euclid. The EAS follows a data-centric approach to data processing, whereby the Data Processing System (DPS) is responsible for the centralized metadata storage and the Distributed Storage System (DSS) supports the distributed storage of data files. The EAS-DPS implements the Euclid Common Data model and along with the EAS-DSS provides numerous services for Euclid Consortium users and SGS subsystems. In addition, the EAS-DPS assists in the preparation of Euclid data releases which are copied to the third EAS subsystem, the ESA developed Science Archive System (SAS) where they become available to the wider astronomical community. The EAS-DPS implements the object-oriented Euclid Common Data Model using a relational DBMS for the storage. The EAS-DPS supports the tracing of the lineage of any data item in the system, provides services for the data quality assessment and the data processing orchestration. The EAS-DSS is a distributed storage system which is based on a set of storage nodes located in each of the ten Science Data Centers of the Euclid SGS. The storage nodes supports a wide range of solutions from local disk, using a unix filesystem, to iRODS nodes or Grid storage elements. In this paper the architectural design of EAS-DPS and EAS-DSS are reviewed: the interaction between them and tests of the already implemented components are described.

Original language	English
Title of host publication	Astronomical Data Analysis Software and Systems XXVI
Editors	Marco Molinaro, Keith Shortridge, Fabio Pasian
Publisher	Astronomical Society of the Pacific
Pages	120-123
ISBN (Electronic)	978-1-58381-930-2
ISBN (Print)	978-1-58381-929-6
Publication status	Published - Nov-2019
Event	Astronomical Data Analysis Software and Systems XXVI - Trieste, Italy Duration: 16-Oct-2016 → 20-Oct-2016

Publication series

Name	ASP Conference Series
Volume	521

Conference

Conference	Astronomical Data Analysis Software and Systems XXVI
Country/Territory	Italy
City	Trieste
Period	16/10/2016 → 20/10/2016

Access to Document

The Role of the Euclid Archive System in the Processing of Euclid and External DataFinal publisher's version, 1.71 MBLicence: Unspecified

http://adsabs.harvard.edu/abs/2019ASPC..521..120W

Handle.net

Persistent link

Cite this

Williams, O. R., Begeman, K., Boxhoorn, D., Droge, B., Tsyganov, A., McFarland, J. P., Valentijn, E. A., Vriend, W. J., & Dabin, C. (2019). The Role of the Euclid Archive System in the Processing of Euclid and External Data. In M. Molinaro, K. Shortridge, & F. Pasian (Eds.), Astronomical Data Analysis Software and Systems XXVI (pp. 120-123). (ASP Conference Series; Vol. 521). Astronomical Society of the Pacific. http://adsabs.harvard.edu/abs/2019ASPC..521..120W

@inbook{4788a49d032444b38804931eaff99a84,

title = "The Role of the Euclid Archive System in the Processing of Euclid and External Data",

abstract = "Euclid is an ESA M2 mission which will create a 15,000 square degrees space-based survey: the Euclid Archive System (EAS) is a core element of the Science Ground Segment (SGS) of Euclid. The EAS follows a data-centric approach to data processing, whereby the Data Processing System (DPS) is responsible for the centralized metadata storage and the Distributed Storage System (DSS) supports the distributed storage of data files. The EAS-DPS implements the Euclid Common Data model and along with the EAS-DSS provides numerous services for Euclid Consortium users and SGS subsystems. In addition, the EAS-DPS assists in the preparation of Euclid data releases which are copied to the third EAS subsystem, the ESA developed Science Archive System (SAS) where they become available to the wider astronomical community. The EAS-DPS implements the object-oriented Euclid Common Data Model using a relational DBMS for the storage. The EAS-DPS supports the tracing of the lineage of any data item in the system, provides services for the data quality assessment and the data processing orchestration. The EAS-DSS is a distributed storage system which is based on a set of storage nodes located in each of the ten Science Data Centers of the Euclid SGS. The storage nodes supports a wide range of solutions from local disk, using a unix filesystem, to iRODS nodes or Grid storage elements. In this paper the architectural design of EAS-DPS and EAS-DSS are reviewed: the interaction between them and tests of the already implemented components are described.",

author = "Williams, {O. R.} and K. Begeman and D. Boxhoorn and B. Droge and A. Tsyganov and McFarland, {J. P.} and Valentijn, {E. A.} and Vriend, {W. J.} and C. Dabin",

year = "2019",

month = nov,

language = "English",

isbn = "978-1-58381-929-6",

series = "ASP Conference Series",

publisher = "Astronomical Society of the Pacific",

pages = "120--123",

editor = "Marco Molinaro and Keith Shortridge and Fabio Pasian",

booktitle = "Astronomical Data Analysis Software and Systems XXVI",

note = "Astronomical Data Analysis Software and Systems XXVI<br/> ; Conference date: 16-10-2016 Through 20-10-2016",

}

Williams, OR, Begeman, K, Boxhoorn, D , Droge, B , Tsyganov, A , McFarland, JP , Valentijn, EA , Vriend, WJ & Dabin, C 2019, The Role of the Euclid Archive System in the Processing of Euclid and External Data. in M Molinaro, K Shortridge & F Pasian (eds), Astronomical Data Analysis Software and Systems XXVI . ASP Conference Series, vol. 521, Astronomical Society of the Pacific, pp. 120-123, Astronomical Data Analysis Software and Systems XXVI
, Trieste, Italy, 16/10/2016. <http://adsabs.harvard.edu/abs/2019ASPC..521..120W>

The Role of the Euclid Archive System in the Processing of Euclid and External Data. / Williams, O. R.; Begeman, K.; Boxhoorn, D. et al.
Astronomical Data Analysis Software and Systems XXVI . ed. / Marco Molinaro; Keith Shortridge; Fabio Pasian. Astronomical Society of the Pacific, 2019. p. 120-123 (ASP Conference Series; Vol. 521).

Research output: Chapter in Book/Report/Conference proceeding › Chapter › Academic › peer-review

TY - CHAP

T1 - The Role of the Euclid Archive System in the Processing of Euclid and External Data

AU - Williams, O. R.

AU - Begeman, K.

AU - Boxhoorn, D.

AU - Droge, B.

AU - Tsyganov, A.

AU - McFarland, J. P.

AU - Valentijn, E. A.

AU - Vriend, W. J.

AU - Dabin, C.

PY - 2019/11

Y1 - 2019/11

N2 - Euclid is an ESA M2 mission which will create a 15,000 square degrees space-based survey: the Euclid Archive System (EAS) is a core element of the Science Ground Segment (SGS) of Euclid. The EAS follows a data-centric approach to data processing, whereby the Data Processing System (DPS) is responsible for the centralized metadata storage and the Distributed Storage System (DSS) supports the distributed storage of data files. The EAS-DPS implements the Euclid Common Data model and along with the EAS-DSS provides numerous services for Euclid Consortium users and SGS subsystems. In addition, the EAS-DPS assists in the preparation of Euclid data releases which are copied to the third EAS subsystem, the ESA developed Science Archive System (SAS) where they become available to the wider astronomical community. The EAS-DPS implements the object-oriented Euclid Common Data Model using a relational DBMS for the storage. The EAS-DPS supports the tracing of the lineage of any data item in the system, provides services for the data quality assessment and the data processing orchestration. The EAS-DSS is a distributed storage system which is based on a set of storage nodes located in each of the ten Science Data Centers of the Euclid SGS. The storage nodes supports a wide range of solutions from local disk, using a unix filesystem, to iRODS nodes or Grid storage elements. In this paper the architectural design of EAS-DPS and EAS-DSS are reviewed: the interaction between them and tests of the already implemented components are described.

AB - Euclid is an ESA M2 mission which will create a 15,000 square degrees space-based survey: the Euclid Archive System (EAS) is a core element of the Science Ground Segment (SGS) of Euclid. The EAS follows a data-centric approach to data processing, whereby the Data Processing System (DPS) is responsible for the centralized metadata storage and the Distributed Storage System (DSS) supports the distributed storage of data files. The EAS-DPS implements the Euclid Common Data model and along with the EAS-DSS provides numerous services for Euclid Consortium users and SGS subsystems. In addition, the EAS-DPS assists in the preparation of Euclid data releases which are copied to the third EAS subsystem, the ESA developed Science Archive System (SAS) where they become available to the wider astronomical community. The EAS-DPS implements the object-oriented Euclid Common Data Model using a relational DBMS for the storage. The EAS-DPS supports the tracing of the lineage of any data item in the system, provides services for the data quality assessment and the data processing orchestration. The EAS-DSS is a distributed storage system which is based on a set of storage nodes located in each of the ten Science Data Centers of the Euclid SGS. The storage nodes supports a wide range of solutions from local disk, using a unix filesystem, to iRODS nodes or Grid storage elements. In this paper the architectural design of EAS-DPS and EAS-DSS are reviewed: the interaction between them and tests of the already implemented components are described.

M3 - Chapter

SN - 978-1-58381-929-6

T3 - ASP Conference Series

SP - 120

EP - 123

BT - Astronomical Data Analysis Software and Systems XXVI

A2 - Molinaro, Marco

A2 - Shortridge, Keith

A2 - Pasian, Fabio

PB - Astronomical Society of the Pacific

T2 - Astronomical Data Analysis Software and Systems XXVI<br/>

Y2 - 16 October 2016 through 20 October 2016

ER -

The Role of the Euclid Archive System in the Processing of Euclid and External Data

Abstract

Publication series

Conference

Access to Document

Handle.net

Fingerprint

Cite this