Commits · 94c0ad6fad47f603a5a52f42503df3e16db9e6a5 · novelai-storage / Basedformer

15 Jul, 2022 6 commits
- Set default to `2049` and correctly calculate the tokens per second. · 94c0ad6f
  Wes Brown authored Jul 14, 2022
  
  94c0ad6f
- Fix shuffling and use provided context size on CLI. · 53bcc538
  Wes Brown authored Jul 14, 2022
  
  53bcc538
- Merge pull request #9 from NovelAI/os.changes · 6c1a2d67
  Eren Doğan authored Jul 14, 2022
  
  6c1a2d67
- Fix passing step to wandb. · 9fc1cc21
  Wes Brown authored Jul 14, 2022
  
  9fc1cc21
- Actually use the `lm_path` config option. · b4bf88c8
  Wes Brown authored Jul 14, 2022
  
  b4bf88c8
- Small fixes. · aa35ad92
  Wes Brown authored Jul 14, 2022
  
  aa35ad92
14 Jul, 2022 4 commits
- Revert mostly to `x=` assignment form. · 8b26deda
  Wes Brown authored Jul 13, 2022
  
  8b26deda
- Add argument handling, a closured `hypernetwork_saver`, and save the final result. · 8073ccfc
  Wes Brown authored Jul 13, 2022
  
  8073ccfc
- Clean up, better reporting. · 704947b4
  Wes Brown authored Jul 13, 2022
  
  704947b4
- Some cleanup, device agnosticism. · cc02ad48
  Wes Brown authored Jul 13, 2022
  
  cc02ad48
08 Jul, 2022 5 commits
- Merge pull request #8 from NovelAI/datasetfixes · 91470a5a
  Eren Doğan authored Jul 08, 2022
  
  91470a5a
- Don't check if identity is dict · b1e335ec
  novelailab authored Jul 08, 2022
  
  b1e335ec
- Danbooru id as identity · dad308db
  novelailab authored Jul 08, 2022
  
  dad308db
- Revert open() changes, fix and switch to build() · 7b770017
  novelailab authored Jul 08, 2022
  
  7b770017
- ImageDataset fixes, scrape and benchmark scripts · 8a038aec
  novelailab authored Jul 08, 2022
  
  8a038aec
07 Jul, 2022 4 commits
- fucked up · 0c96e233
  Eren Doğan authored Jul 06, 2022
  
  0c96e233
- more checks · 7e9cc877
  Eren Doğan authored Jul 06, 2022
  
  7e9cc877
- more dataset stuff · 690dd2cf
  Eren Doğan authored Jul 06, 2022
  
  690dd2cf
- zero2 works · fb134d28
  novelailab authored Jul 06, 2022
  
  fb134d28
06 Jul, 2022 4 commits
- experimental fused softmax support · 9d27a5cc
  novelailab authored Jul 06, 2022
  
  9d27a5cc
- add fusedad, fix bug in hypertrain, add setup pkgs · 7e2ee30f
  novelailab authored Jul 05, 2022
  
  7e2ee30f
- update hypetrain with API changes · 94ad5ad7
  novelailab authored Jul 05, 2022
  
  94ad5ad7
- push image fixes and builder example(not done) · 5a4b10c7
  novelailab authored Jul 05, 2022
  
  5a4b10c7
05 Jul, 2022 17 commits
- Merge pull request #7 from NovelAI/fiareader · cf0224e4
  Eren Doğan authored Jul 05, 2022
```
mmap based FIAReader
```
  cf0224e4
- delete redundant init.py · da4bc5ad
  novelailab authored Jul 05, 2022
  
  da4bc5ad
- metadata=False by default · 7cf62ce3
  novelailab authored Jul 05, 2022
  
  7cf62ce3
- get_metadata, from id · ae114e18
  novelailab authored Jul 05, 2022
  
  ae114e18
- read_from_id to read from danbooru id · dff538a0
  novelailab authored Jul 05, 2022
  
  dff538a0
- change metadata -> index in ShardedImageDataset · 0f8be39c
  novelailab authored Jul 04, 2022
  
  0f8be39c
- make so can transfer to cuda after reduced · cb67abcb
  novelailab authored Jul 04, 2022
  
  cb67abcb
- add metadata and better error handling · 5c846daf
  novelailab authored Jul 04, 2022
  
  5c846daf
- more properties · 1a66f013
  novelailab authored Jul 04, 2022
  
  1a66f013
- return ids for real · 4577aa6f
  novelailab authored Jul 04, 2022
  
  4577aa6f
- make the permute faster by doing it on batch · 8056efa1
  novelailab authored Jul 04, 2022
  
  8056efa1
- return identity · df7a70ab
  novelailab authored Jul 04, 2022
  
  df7a70ab
- add more stuff · a0b66a4b
  novelailab authored Jul 04, 2022
  
  a0b66a4b
- ImageDatasetBuilder · 2b10353c
  novelailab authored Jul 04, 2022
  
  2b10353c
- add transform support and stacking · 176b21d8
  Eren Doğan authored Jul 04, 2022
  
  176b21d8
- make sure to wait for the executor map futures to finish · 7f9ceee2
  Eren Doğan authored Jul 04, 2022
  
  7f9ceee2
- fix __len__ · 0520ae59
  Eren Doğan authored Jul 04, 2022
  
  0520ae59