Check the docs to see the options you have for updating a Key Vault. One feature will let you block all external access to the Key Vault.
Need more? Here's the solution.