Running the SkyMapper Science Data Pipeline: To be a Big Fish in a Small Pond, or a Small Fish in a Big Ocean?

被引：0

作者：

Luvaul, Lance C. ^{[1
]}

Onken, Christopher A. ^{[1
]}

Wolf, Christian ^{[1
]}

Smillie, Jonathan G. ^{[2
]}

Sebo, Kim M. ^{[1
]}

机构：

[1] Australian Natl Univ, Res Sch Astron & Astrophys, Canberra, ACT 2611, Australia

[2] Australian Natl Univ, Natl Comp Infrastruct, Canberra, ACT 2601, Australia

来源：

ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XXV | 2017年 / 512卷

关键词：

D O I：

暂无

中图分类号：

P1 [天文学];

学科分类号：

0704 ;

摘要：

We review structure and frameworks behind the SkyMapper Science Data Pipeline (SDP), and consider the challenges of deploying on two disparate platforms: a publicly shared, massively parallel, queue-scheduled compute fabric, and a dedicated NUMA-based, multi-core, mini-supercomputer. Concepts reviewed include a) how to impose a layer of central operator control over hundreds of jobs of varying type and CPU/IO profile, all running concurrently and at different stages in their logic, b) how to maintain configuration control in an ever-changing algorithmic environment while not giving up ease of build and deployment, and c) how to configure a NUMA-architected machine for optimal cache buffer usage, process-to-memory locality, and user/system CPU cycle ratio.

引用

页码：393 / 397

页数：5

共 50 条

[31] Zebrafish physiology: big interest in a small fish
Zetterberg, Henrik
ACTA PHYSIOLOGICA, 2011, 201 (03) : 295 - 296
[32] Genetic discontinuity of big fish in a small sea
Reeb, Carol A.
PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA, 2010, 107 (06) : 2377 - 2378
[33] Big fish in a big pond: Tools and techniques to make a large organic chemistry class feel like a small one
Keirstead, Amy
ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL SOCIETY, 2018, 255
[34] Private health care legislations - Big fish eating small fish
George, A
ECONOMIC AND POLITICAL WEEKLY, 1998, 33 (45) : 2833 - 2834
[35] Big fish eat small fish: implications for food chain length?
U. Sommer
E. Charalampous
M. Scotti
M. Moustaka-Gouni
Community Ecology, 2018, 19 : 107 - 115
[36] Small Fish Become Big Fish: Mergers in Stackelberg Markets Revisited
Brito, Duarte
Catalao-Lopes, Margarida
B E JOURNAL OF ECONOMIC ANALYSIS & POLICY, 2011, 11 (01):
[37] Big fish eat small fish: implications for food chain length?
Sommer, U.
Charalampous, E.
Scotti, M.
Moustaka-Gouni, M.
COMMUNITY ECOLOGY, 2018, 19 (02) : 107 - 115
[38] Big Fish and Small Fish Debate-An Examination of the Prosecutorial Discretion
Takemura, Hitomi
INTERNATIONAL CRIMINAL LAW REVIEW, 2007, 7 (04) : 677 - 685
[39] A small fish with a big future: zebrafish in behavioral neuroscience
Gerlai, Robert
REVIEWS IN THE NEUROSCIENCES, 2011, 22 (01) : 3 - 4
[40] Acoustic analysis of big ocean data to monitor fish sounds
Sattar, Farook
Cullis-Suzuki, Sarika
Jin, Feng
ECOLOGICAL INFORMATICS, 2016, 34 : 102 - 107

← 1 2 3 4 5 →