In the dial-a-ride problem, DARP for short, given a set of requests on pick up and delivery, we aim at finding a minimum-cost set of vehicle routes that meets all the requests. In this study, we focus on users' inconvenience costs, and propose a problem of minimizing them in order to achieve fairness. We call the resulting problem DARP-F. We demonstrate that DARP-F has considerable advantages over the conventional one using time-window constraints, DARP-T for short. In particular, DARP-T is infeasible in the presence of requests that conflict each other, while DARP-F is always feasible and offers a best compromise solution taking the inconvenience costs into account. We discuss how the inconvenience costs should be defined and be treated in practice through numerical experiments. In addition, we show that DARP-F is NP-hard, and give an integer programming model, which can solve middle-sized instances in reasonable computation time.