Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Sign in / Register
Toggle navigation
B
Basedformer
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Locked Files
Issues
0
Issues
0
List
Boards
Labels
Service Desk
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Security & Compliance
Security & Compliance
Dependency List
License Compliance
Packages
Packages
List
Container Registry
Analytics
Analytics
CI / CD
Code Review
Insights
Issues
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
novelai-storage
Basedformer
Commits
b07251f0
Commit
b07251f0
authored
Mar 20, 2022
by
novelailab
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
add todos and some stuff
parent
d9df990b
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
9 additions
and
3 deletions
+9
-3
train.py
train.py
+9
-3
No files found.
train.py
View file @
b07251f0
...
...
@@ -24,9 +24,12 @@ model_config = {
# we need 250 batch size to train the small GPT.
train_config
=
{
"data_path"
:
"/home/xuser/diffusionstorage/datasets/OWT2-gpt2-full.map"
,
"save_path"
:
"/home/xuser/diffusionstorage/workspace/kuru/basedformer/models/owt2train"
,
"run_name"
:
"owt2-125m"
,
"lr"
:
6e-4
,
"end_lr"
:
6e-4
,
"warmup_steps"
:
2
0
,
"warmup_steps"
:
5
0
,
"bs"
:
16
,
"gas"
:
16
,
"seed"
:
69
,
...
...
@@ -35,9 +38,12 @@ bs = train_config["bs"]
gas
=
train_config
[
"gas"
]
model
=
GPTModel
.
neox_init
(
model_config
)
.
cuda
()
.
bfloat16
()
opt
=
optimizer
.
BasedOptimizer
(
model
.
parameters
(),
train_config
,
"adamw"
)
train_dataset
=
utils
.
FbDataset
(
2049
,
"/home/xuser/diffusionstorage/datasets/sigurd/map/sigurd_v5_2049.map"
)
# TODO: Add load, add evals, add FP16 AMP, and Data Parallel.
train_dataset
=
utils
.
FbDataset
(
2049
,
train_config
[
"data_path"
])
train_loader
=
data
.
DataLoader
(
train_dataset
,
batch_size
=
bs
*
gas
,
shuffle
=
False
,
num_workers
=
0
)
wandb
.
init
(
project
=
"basedformer-tests"
,
name
=
"sigurd_v5_2049"
)
wandb
.
init
(
project
=
"basedformer-tests"
,
name
=
train_config
[
"run_name"
],
config
=
{
**
train_config
,
**
model_config
}
)
t
=
tqdm
(
train_loader
)
for
input_ids
,
labels
in
t
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment