In the real world, listeners often need to track multiple simultaneous sources in order to maintain awareness of the relevant sounds in their environments. Thus, there is reason to believe that simple single source sound localization tasks may not accurately capture the impact that a listening device such as a hearing aid might have on a listener's level of auditory awareness. In this experiment, 10 normal hearing listeners and 20 hearing impaired listeners were tested in a task that required them to identify and localize sound sources in three different listening tasks of increasing complexity: a single-source localization task, where listeners identified and localized a single sound source presented in isolation; an added source task, where listeners identified and localized a source that was added to an existing auditory scene, and a remove source task, where listeners identified and localized a source that was removed from an existing auditory scene. Hearing impaired listeners completed these tasks with and without the use of their previously fit hearing aids. As expected, the results show that performance decreased both with increasing task complexity and with the number of competing sound sources in the acoustic scene. The results also show that the added source task was as sensitive to differences in performance across listening conditions as the standard localization task, but that it correlated with a different pattern of subjective and objective performance measures across listeners. This result suggests that a measure of complex auditory situation awareness such as the one tested here may be a useful tool for evaluating differences in performance across different types of listening devices, such as hearing aids or hearing protection devices.