OK, I think that is the “problem” so to speak. At that size, it means you have an image about 750×750 pixels which is not a lot considering the number of curves you have on that edge. And since pixels are square, I suspect that when you make the selection (before even expanding it), the selection is “straight” for a few pixels where you would expect a curve. Can you check that?
One possible solution would be to increase the resolution. Go to Image > Resize. Select By Print Size, and there, change the Resolution to 300 pixels/inch. Now, you will have twice as many pixels to work with as your work will become a 1500×1500 pixels image. At this point, you can try the same manipulation to select and expand. You might have a better result.
Try that and let us know.