Surprise Benchmarking: The Why, What, and How

被引：0

作者：

Benson, Lawrence ^{[1
]}

Binnig, Carsten ^{[2
,3
]}

Bodensohn, Jan-Micha ^{[2
,3
]}

Lorenzi, Federico ^{[4
]}

Luo, Jigao ^{[2
,3
]}

Porobic, Danica ^{[5
]}

Rabl, Tilmann ^{[6
]}

Sanghi, Anupam ^{[2
]}

Sears, Russell ^{[7
]}

Tozun, Pinar ^{[8
]}

Ziegler, Tobias ^{[2
]}

机构：

[1] Tech Univ Munich, Munich, Germany

[2] Tech Univ Darmstadt, Darmstadt, Germany

[3] DFKI, Kaiserslautern, Germany

[4] TigerBeetle, San Francisco, CA USA

[5] Oracle, Santa Clara, CA USA

[6] Hasso Plattner Inst, Potsdam, Germany

[7] CrystalDB, San Francisco, CA USA

[8] IT Univ Copenhagen, Copenhagen, Denmark

来源：

PROCEEDINGS OF THE 2024 10TH INTERNATIONAL WORKSHOP ON TESTING DATABASE SYSTEMS, DBTEST 2024 | 2024年

关键词：

benchmarking; database;

D O I：

10.1145/3662165.3662763

中图分类号：

TP3 [计算技术、计算机技术];

学科分类号：

0812 ;

摘要：

Standardized benchmarks are crucial to ensure a fair comparison of performance across systems. While extremely valuable, these benchmarks all use a setup where the workload is well-de.ned and known in advance. Unfortunately, this has led to overly-tuning data management systems for particular benchmark workloads such as TPC-H or TPC-C. As a result, benchmarking results frequently do not re.ect the behavior of these systems in many real-world settings since workloads often signi.cantly vary from the "known" benchmarking workloads. To address this issue, we present surprise benchmarking, a complementary approach to the current standardized benchmarking where "unknown" queries are exercised during the evaluation.

引用

页码：1 / 8

页数：8

共 50 条

[1] What happened to the Soviet Union? How and why American Sovietologists were caught by surprise
Kramer, M
[J]. POLITICAL SCIENCE QUARTERLY, 2003, 118 (04) : 688 - 690
[2] What happened to the Soviet Union? How and why American Sovietologists were caught by surprise
Noonan, NC
[J]. RUSSIAN REVIEW, 2003, 62 (03): : 480 - 481
[3] Thoughts on benchmarking of function modeling: Why and how
Bohm, Matthew
Eckert, Claudia
Sen, Chiradeep
Srinivasan, Venkatamaran
Summers, Joshua D.
Vermaas, Pieter
[J]. AI EDAM-ARTIFICIAL INTELLIGENCE FOR ENGINEERING DESIGN ANALYSIS AND MANUFACTURING, 2017, 31 (04): : 393 - 400
[4] HOW TO, WHAT, AND WHY
LINVILLE, LB
[J]. AMERICAN BAR ASSOCIATION JOURNAL, 1982, 68 (NOV): : 1344 - 1344
[5] Why the surprise?
Hogan, David B.
[J]. CANADIAN MEDICAL ASSOCIATION JOURNAL, 2006, 174 (12) : 1746 - 1746
[6] WHY THE SURPRISE
PYKETT, C
[J]. NEW SCIENTIST, 1994, 143 (1940) : 45 - 45
[7] ADVERTISING - WHY, WHAT, AND HOW
EMERY, J
[J]. JOURNAL OF CATARACT AND REFRACTIVE SURGERY, 1989, 15 (03): : 251 - 252
[8] Femtocell: What, Why, and How?
Raza, Ali
Bhutto, Zuhaibuddin
Kumar, Ramesh
Memon, Mudasar Latif
Shaikh, Murtaza Hussain
Shah, Jalal
Idrees, Muhammad
[J]. INTERNATIONAL JOURNAL OF COMPUTER SCIENCE AND NETWORK SECURITY, 2019, 19 (07): : 215 - 218
[9] The why, how, and what of mentoring
Trubowitz, S
[J]. PHI DELTA KAPPAN, 2004, 86 (01) : 59 - 62
[10] SCTP: What, Why, and How
Natarajan, Preethi
Baker, Fred
Amer, Paul D.
Leighton, Jonathan T.
[J]. IEEE INTERNET COMPUTING, 2009, 13 (05) : 81 - 85

← 1 2 3 4 5 →