I might have found the reason for slow performance. On another workbook (open in background) I had a Linked Picture with reference to a dynamic range. When I remove the link the performance increased dramatically. After removing the Linked Picture my refresh time seams to be stable at 40-45 seconds. This is more in line with expectations.
Does anyone know how a linked picture in another workbook can disturb EPM performance?
