disable torch weight initialization and CLIP downloading/reading checkpoint to speedup creating sd model from config