Pirkko, depending on the tool you use, you will get rough edges.
- If you use a Selection tool (like the Magic Wand), make sure the Anti-alias is checked. Even if it is, it might not be smooth.
- If you can Feather your selection by 1 pixel, it might help smooth the edges somewhat.
- If you can use a vector shape (if the design is a Preset Shape), then you can go to Selections > From Vector Object, and that will give the best edges possible.
- And of course, the smaller the selection is, the rougher it will be.
