The purpose of image smoothing is to process the image through certain technical methods, so as to remove the texture details in the image while preserving the important structural edges. Therefore, how to distinguish the two correctly has become the key to image smoothing. Gradients, as an important index for calculating the speed of image change, are an effective measure to distinguish the structural edges and texture details. However, the gradient difference between texture details and structural edges in different images or different regions of the same image is not fixed. In order to effectively distinguish structural edges and texture details based on gradients, an image smoothing method was proposed based on image decomposition and relative total variation. To expand the difference between the structural edges and texture details, the gradients of texture details were reduced without changing the gradients of structural edges as much as possible. The image was decomposed in frequency domain under the constraint of multi-directional gradient, and then the smooth components of the decomposed image were extracted. Next, for the smooth component of the input image, at the specific scale, based on the structural differences of a specific size region of the image, the relative total variation method was employed to remove the texture details at this scale while preserving the structure edges. Finally, through iterative optimization, the size of the image region was continuously adjusted to gradually remove texture details of different scales. Compared with the existing algorithms, the new method could attain better visual effects in effectively removing the texture details and completely preserving the structure edges.