In 2012 Pennsylvania's legislature increased the unconventional natural gas (UNG) well-to-building setback requirement from 200 ft to 500 ft through Act 13. To evaluate this policy, we identified all setback incident locations where a UNG well was within 500 ft of a building both before and after the implementation of Act 13. Using an interrupted time series design, we found that Act 13 did not significantly alter how wells were sited in relation to nearby buildings. Of the 1,042 wells that contained a building within 500 ft—equating to ~10.1% of UNG wells (n = 11,148) and ~14.7% well pads (n = 479)—a total of 371 well setback incidents occurred after Act 13, likely due from the existing well pad exemption (35%) and a combination of landowner consent and regulatory variances rather than encroaching building construction. Overall, our study suggests that exemptions are an important and underappreciated aspect of oil and gas well setback rulemaking and highlights the relevance of other health-protective regulatory tools often promulgated alongside setbacks. New or amended setback regulations should revisit exemption procedures and where warranted, impose additional mitigation measures to ensure setback regulations provide adequate protections for health and safety as intended.