Our aim is to obtain efficient algorithms for image regularization optimized for removing different types of noise. To accomplish this, we combine total variation regularization with a noise-specific way to measure the fidelity between the noisy image and the reconstruction. We find a minimum of the resulting functional with a quasi-Newton method, which converges faster than the common method of gradient descent. As examples we consider Poisson noise and impulse noise. We prove convergence of the algorithm for a large class of fidelity terms.