The re-use of recycled aggregates to produce concrete is an important step forward to sustainable construction. Nonetheless, it is necessary to examine the structures constructed with recycled aggregate when subjected to various conditions, especially when exposed to high temperature. Hence, this experimental research determines the influence of elevated temperature on the physical and mechanical properties of concrete made by replacing 20% of Natural Coarse Aggregate (NCA) by recycled Refractory Brick Aggregate (RBA). Two different types of RBA were used to manufacture concrete mixture with three water/cement ratios (w/c = 0.59, 0.47 and 0.38). Specimens were heated with a rate of 3 degrees C/min from 20 degrees C to 800 degrees C, then, they were maintained for 1 h (when reaching the desired temperature) and naturally cooled to room temperature. Afterwards, tests were conducted on concrete made with RBA to evaluate density, porosity, mass loss, change of volume, ultrasonic pulse velocity, damage degree, compressive strength and dynamic elastic modulus before and after heating. These properties were compared to those obtained on conventional concretes made with 100% of NCA. Moreover, a microscopic study was carried out to recognize the change of surface crack. In conclusion, RBA is a potential alternative for NCA in concrete manufacturing and can help to remain the concrete properties after heating.