Evaluating attributed information retrieval (AIR) systems requires assessing both informativeness and attributability. To enable scalable evaluation, LLM-sourced ground truth data is frequently used, yet the validity of this practice remains unclear. We replicate the evaluation framework of Djeddal et al. [1] which relies on LLM-written ground truth answers, and additionally crowdsource human-written answers and pairwise preference judgments. This allows us to investigate (1) how robust reference-based evaluation measures are to gold reference variation; (2) to what extent do LLM judges agree with human annotators; and (3) which automatic measures best predict human and LLM preferences? Our findings reveal substantial sensitivity of reference-based measures to gold reference choice, and human and LLM judges exhibiting low agreement on preference judgments, despite similar aggregate tendencies. Furthermore, no automatic evaluation measure strongly predicts human preferences, suggesting a fundamental methodological shortcoming in current AIR evaluation practices.