Finding and Using Edges to Control Sharpening in Photoshop

Sometimes even the tonal range controls of the Smart Sharpen tool don’t apply the sharpening to just where you want. In this case you can use selections and masks along with High Pass, Maximum, or Minimum filters to help you selectively apply your sharpening.

1. With the Leaves in Ice_RGB_Sharpen image still open, turn off the visibility of all the sharpening layers you have created. Then create a new duplicate layer of the original Background image. Name the layer High Pass.
2. Active the Channel panel and duplicate the channel that contains the image elements you would like to sharpen most. Here the leaves and some of the ice edges stand out in highest contrast; here it is the Red Channel. This will create a fourth channel named Red Channel Copy. Now, the following two steps can all be perfomed on this new fifth channel, but I will create separate channels just to keep the steps straight.
3. Duplicate the Red Channel Copy to create a fifth channel, and name this Find Edges. Then Choose Filter --> Stylize --> Find Edges. This will create a channel with high contrast edges.
4. Again Duplicate this channel to create a sixth channel, and label it Find Edges_Invert. Then choose Image --> Adjustments --> Invert (Cmd+I/Ctrl+I) to reverse or invert the black and white values (create a negative image). This step is done to convert the high contrast edges into white values that can be loaded as a selection.
5. Once again duplicate this Find Edges inverted channel and name it High Pass. Then choose Filter --> Other --> High Pass to activate its dialog. The radius value you choose here will depend upon the contents of your image and the current channel. Experiment with values of 0.1, 1.0, and 10.0 and apply the Radius = 10. Add this value to the name of the channel. The white areas in the channel are a mask that will be converted into a selection when you load this channel. This is why you had to Invert the channel to convert the edges into white. With this in mind you can edit this channel to adjust the white areas to suit your selection desires. 
6. One way to edit a mask channel is to use the Levels dialog. Choose Image --> Adjust --> Levels to activate the dialog. Adjust the highlight, shadow, and midtone sliders to control the distribution of black and white. Note that the intermediate gray areas will   load are partially selected areas, creating smooth gradational edges between the full sharpened (white) and unsharpened (black) areas of the selection 
7. Duplicate your original Find Edges inverted channel (here channel 6). Name this Maximum. With this channel active choose Filter --> Other --> Maximum to activate the dialog. Assign a Radius = 3.
8. Now choose Filters --> Blur --> Gaussian Blur and assign a Radius = 3 to widen and soften the white soon-to-be selection edges. Rename this channel (here #9) to reflect these alterations. Keeping track of your adjustments via labeling makes it easier to   remember, and repeat, what you have done.
9. Now to use the last channel mask to control your sharpening, First activate your Maximum image layer. Then choose Select --> Load Selection --> Maximum; to accomplish this with a keyboard shortcut, use Cmd+Opt+9/Ctrl+Alt+9 (the channel number). This loads the Maximum channel as a selection.
10. To be able to see the impact of the sharpening you are about to apply, type Cmd+H/Ctrl+H to hide the selection. Then choose Filter --> Sharpen --> Unsharp Mask or Smart Sharpen (here I used Smart Sharpen) to apply the sharpening to your loaded selection. After you have some practice visualizing the white areas in a channel mask as a selection, this process and your value selections for creating and editing your masks will become much easier!