Summary | When performing loss assessment of a geographically dispersed building portfolio, the response or loss (fragility or vulnerability) function of any given archetype building is typically considered to be a consistent property of the building itself. On the other hand, recent advances in record selection have shown that the seismic response of a structure is, in general, dependent on the nature of the hazard at the site of interest. This apparent contradiction begs the question: Are building fragility and vulnerability functions independent of site, and if not, what can be done to avoid having to reassess them for each site of interest? In the following, we show that there is a non‐negligible influence of the site, the degree of which depends on the intensity measure adopted for assessment. Employing a single‐period (e.g., first‐mode), spectral acceleration would require careful record selection at each site and result to significant site‐to‐site variability of the fragility or vulnerability function. On the other hand, an intensity measure comprising the geometric mean of multiple spectral accelerations considerably reduces such variability. In tandem with a conditional spectrum record selection that accounts for multiple sites, it can offer a viable approach for incorporating the effect of site dependence into fragility and vulnerability estimates. Copyright © 2017 John Wiley & Sons, Ltd.