This depends on the type of error that has occurred. If the error reported is during the upload process or because you are attempting to upload an unsupported format, there is no charge. Otherwise, you will only be charged for the source video encoding bandwidth and not for the output files that have resulted in error.
Knowledge Base
Find your answers here.