Running the SkyMapper Science Data Pipeline: To be a Big Fish in a Small Pond, or a Small Fish in a Big Ocean?

被引：0

作者：

Luvaul, Lance C. ^{[1
]}

Onken, Christopher A. ^{[1
]}

Wolf, Christian ^{[1
]}

Smillie, Jonathan G. ^{[2
]}

Sebo, Kim M. ^{[1
]}

机构：

[1] Australian Natl Univ, Res Sch Astron & Astrophys, Canberra, ACT 2611, Australia

[2] Australian Natl Univ, Natl Comp Infrastruct, Canberra, ACT 2601, Australia

来源：

ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XXV | 2017年 / 512卷

关键词：

D O I：

暂无

中图分类号：

P1 [天文学];

学科分类号：

0704 ;

摘要：

We review structure and frameworks behind the SkyMapper Science Data Pipeline (SDP), and consider the challenges of deploying on two disparate platforms: a publicly shared, massively parallel, queue-scheduled compute fabric, and a dedicated NUMA-based, multi-core, mini-supercomputer. Concepts reviewed include a) how to impose a layer of central operator control over hundreds of jobs of varying type and CPU/IO profile, all running concurrently and at different stages in their logic, b) how to maintain configuration control in an ever-changing algorithmic environment while not giving up ease of build and deployment, and c) how to configure a NUMA-architected machine for optimal cache buffer usage, process-to-memory locality, and user/system CPU cycle ratio.

引用

页码：393 / 397

页数：5

共 50 条

[1] Small fish in a big pond
Ollier, W
Worthington, J
BRITISH JOURNAL OF RHEUMATOLOGY, 1997, 36 (09): : 931 - 932
[2] Big FISH, small pond
Wong, J
NATURE MEDICINE, 1996, 2 (12) : 1285 - 1285
[3] Small Fish Big Pond
Skinner, Jonathan
ANGELAKI-JOURNAL OF THE THEORETICAL HUMANITIES, 2009, 14 (02): : 111 - 113
[4] Small Fish in a Big Pond or Big Fish in a Small Pond? The Effects of Educational Mismatch on Subjective Wellbeing
Artes, Joaquin
del Mar Salinas-Jimenez, Maria
Salinas-Jimenez, Javier
SOCIAL INDICATORS RESEARCH, 2014, 119 (02) : 771 - 789
[5] Small Fish in a Big Pond or Big Fish in a Small Pond? The Effects of Educational Mismatch on Subjective Wellbeing
Joaquín Artés
Maria del Mar Salinas-Jiménez
Javier Salinas-Jiménez
Social Indicators Research, 2014, 119 : 771 - 789
[6] Small fish, big science
Bradbury, J
PLOS BIOLOGY, 2004, 2 (05) : 568 - 572
[7] Small fish cause a stir in in the big pond
不详
CHEMISTRY IN BRITAIN, 2001, 37 (06) : 13 - 13
[8] Small fish in a big pond: magnitude discrimination in the archer fish
Leibovich, T.
Vasserman, G.
Ben-Simon, A.
Segev, R.
Henik, A.
JOURNAL OF MOLECULAR NEUROSCIENCE, 2012, 48 : S66 - S67
[9] A BIG FISH IN A SMALL POND OR A SMALL FISH IN A BIG POND? IMPORTANCE OF INTRA-VERSUS INTERGROUP STATUS ACROSS CULTURES
Chen, Ya-Ru
STATUS AND GROUPS, 2005, 7 : 43 - 68
[10] Performance-based publisher ratings and the visibility/impact of books: Small fish in a big pond, or big fish in a small pond?
Zuccala, Alesia A.
Polonen, Janne
Guns, Raf
Roeggen, Vidar
Kulczycki, Emanuel
Bruun, Kasper
Savolainen, Eeva
QUANTITATIVE SCIENCE STUDIES, 2021, 2 (02): : 588 - 615

← 1 2 3 4 5 →