Hi,
The billing document has not been released to accounting . Release it to accounting and attempt to cancel it in VF11.
Check your Customer Master - Acct assgmt group whether empty or not. Cross check for the G/L 's in VKOA for the corresponding key.
Regards,
SRK