Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Sign in / Register
Toggle navigation
B
Basedformer
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Locked Files
Issues
0
Issues
0
List
Boards
Labels
Service Desk
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Security & Compliance
Security & Compliance
Dependency List
License Compliance
Packages
Packages
List
Container Registry
Analytics
Analytics
CI / CD
Code Review
Insights
Issues
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
novelai-storage
Basedformer
Commits
6c1a2d67
Commit
6c1a2d67
authored
Jul 14, 2022
by
Eren Doğan
Committed by
GitHub
Jul 14, 2022
Browse files
Options
Browse Files
Download
Plain Diff
Merge pull request #9 from NovelAI/os.changes
parents
91470a5a
9fc1cc21
Changes
4
Expand all
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
224 additions
and
143 deletions
+224
-143
basedformer/dataset.py
basedformer/dataset.py
+3
-4
basedformer/lm_utils.py
basedformer/lm_utils.py
+4
-2
hypertrain.py
hypertrain.py
+211
-137
requirements.txt
requirements.txt
+6
-0
No files found.
basedformer/dataset.py
View file @
6c1a2d67
import
numpy
as
np
import
torch
import
mmap
import
pickle
import
concurrent
from
torch.utils
import
data
from
simplejpeg
import
decode_jpeg
import
simplejpeg
import
pickle
from
pathlib
import
Path
from
PIL
import
Image
from
tqdm
import
tqdm
from
concurrent.futures
import
as_completed
import
requests
...
...
@@ -54,6 +50,9 @@ class ShardedDataset(data.Dataset):
class
ShardedImageDataset
(
data
.
Dataset
):
def
__init__
(
self
,
dataset_path
:
str
,
index_path
:
str
,
metadata_path
=
None
,
threads
=
None
,
inner_transform
=
None
,
outer_transform
=
None
,
skip
=
0
,
bsz
=
256
,
world_size
=
1
,
local_rank
=
0
,
global_rank
=
0
,
device
=
"cpu"
):
from
simplejpeg
import
decode_jpeg
import
simplejpeg
from
PIL
import
Image
self
.
skip
=
skip
self
.
threads
=
threads
...
...
basedformer/lm_utils.py
View file @
6c1a2d67
...
...
@@ -87,11 +87,13 @@ def load_from_path(config_folder=None, strict=False):
model
=
_load_dict_model
(
model_class
,
model_config
,
model_path
,
strict
=
strict
)
return
model
def
_load_dict_model
(
model_class
,
config
,
path
=
None
,
state_dict
=
None
,
strict
=
False
):
def
_load_dict_model
(
model_class
,
config
,
path
=
None
,
state_dict
=
None
,
strict
=
False
,
device
=
"cuda"
):
# I am kinda sad that we will not have a load function in lm object itself.
# might be better to add load functions -- actually nope.
if
path
:
state_dict
=
utils
.
SplitCheckpoint
(
path
,
device
=
"cuda"
)
state_dict
=
utils
.
SplitCheckpoint
(
path
,
device
=
device
)
state_dict
.
device
=
device
model
=
utils
.
no_init
(
lambda
:
model_class
(
config
))
model
.
load_state_dict
(
state_dict
,
strict
=
strict
)
...
...
hypertrain.py
View file @
6c1a2d67
This diff is collapsed.
Click to expand it.
requirements.txt
View file @
6c1a2d67
einops
wandb
transformers
dotmap
icecream
termcolor
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment