DiffML: End-to-end Differentiable ML Pipelines

被引:3
|
作者
Hilprecht, Benjamin [1 ]
Hammacher, Christian [2 ]
Reis, Eduardo [1 ]
Abdelaal, Mohamed
Binnig, Carsten [1 ]
机构
[1] Tech Univ Darmstadt, Darmstadt, Germany
[2] Software AG, Mainz, Germany
关键词
data engineering; differentiable ML pipelines; data cleaning;
D O I
10.1145/3595360.3595857
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
In this paper, we present our vision of differentiable ML pipelines called DiffML that truly allows to automate the construction of ML pipelines in an end-to-end fashion. DiffML allows to jointly train not just the ML model itself but also the entire pipeline including data engineering steps, e.g., data cleaning, data augmentation, etc. Our core idea is to formulate all steps in a differentiable way such that the entire pipeline can be trained using backpropagation. However, this is a non-trivial problem and opens up many new research questions. To show the feasibility of this direction, we demonstrate initial ideas and a general principle of how typical data engineering steps can be formulated as differentiable programs and jointly learned with the ML model. Moreover, we discuss a research roadmap and core challenges that have to be systematically tackled to enable fully differentiable ML pipelines.
引用
收藏
页数:7
相关论文
共 50 条
  • [41] An End-to-End Platform for Managing Third-Party Risks in Oil Pipelines
    Casas, Edmundo
    Ramos, Leo
    Romero, Cristian
    Rivas-Echeverria, Francklin
    Cerpa, Dunetchka
    Hernandez, Pablo
    Orellana, Gonzalo
    Ibarra, Jose Luis
    Albrecht, Carlos Rosas
    Cuevas, Natalia
    Hurtado, Juan Carlos Gallardo
    IEEE ACCESS, 2024, 12 : 77831 - 77851
  • [42] RaLL: End-to-End Radar Localization on Lidar Map Using Differentiable Measurement Model
    Yin, Huan
    Chen, Runjian
    Wang, Yue
    Xiong, Rong
    IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, 2022, 23 (07) : 6737 - 6750
  • [43] End-to-end differentiable blind tip reconstruction for noisy atomic force microscopy images
    Yasuhiro Matsunaga
    Sotaro Fuchigami
    Tomonori Ogane
    Shoji Takada
    Scientific Reports, 13
  • [44] End-to-End Differentiable Learning to HDR Image Synthesis for Multi-exposure Images
    Kim, Jung Hee
    Lee, Siyeong
    Kang, Suk-Ju
    THIRTY-FIFTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THIRTY-THIRD CONFERENCE ON INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE AND THE ELEVENTH SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2021, 35 : 1780 - 1788
  • [45] End-to-end learned single lens design using fast differentiable ray tracing
    Li, Zongling
    Hou, Qingyu
    Wang, Zhipeng
    Tan, Fanjiao
    Liu, Jin
    Zhang, Wei
    OPTICS LETTERS, 2021, 46 (21) : 5453 - 5456
  • [46] End-to-end differentiable blind tip reconstruction for noisy atomic force microscopy images
    Matsunaga, Yasuhiro
    Fuchigami, Sotaro
    Ogane, Tomonori
    Takada, Shoji
    SCIENTIFIC REPORTS, 2023, 13 (01)
  • [47] Attention-Based End-to-End Differentiable Particle Filter for Audio Speaker Tracking
    Zhao, Jinzheng
    Xu, Yong
    Qian, Xinyuan
    Liu, Haohe
    Plumbley, Mark D.
    Wang, Wenwu
    IEEE OPEN JOURNAL OF SIGNAL PROCESSING, 2024, 5 : 449 - 458
  • [48] End-to-End 6-DoF Object Pose Estimation Through Differentiable Rasterization
    Palazzi, Andrea
    Bergamini, Luca
    Calderara, Simone
    Cucchiara, Rita
    COMPUTER VISION - ECCV 2018 WORKSHOPS, PT III, 2019, 11131 : 702 - 715
  • [49] The end of end-to-end?
    Garfinkel, S
    TECHNOLOGY REVIEW, 2003, 106 (06) : 30 - 30
  • [50] End-to-end consensus using end-to-end channels
    Wiesmann, Matthias
    Defago, Xavier
    12TH PACIFIC RIM INTERNATIONAL SYMPOSIUM ON DEPENDABLE COMPUTING, PROCEEDINGS, 2006, : 341 - +