Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Sign in / Register
Toggle navigation
B
Basedformer
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Locked Files
Issues
0
Issues
0
List
Boards
Labels
Service Desk
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Security & Compliance
Security & Compliance
Dependency List
License Compliance
Packages
Packages
List
Container Registry
Analytics
Analytics
CI / CD
Code Review
Insights
Issues
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
novelai-storage
Basedformer
Commits
44751bc6
Commit
44751bc6
authored
May 09, 2022
by
novelailab
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
config almost done
parent
42870e7b
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
8 deletions
+4
-8
basedformer/gptj.py
basedformer/gptj.py
+4
-8
No files found.
basedformer/gptj.py
View file @
44751bc6
from
typing
import
KeysView
from
typing
import
Callable
,
KeysView
from
regex
import
D
from
regex
import
D
import
torch
import
torch
import
torch.nn
as
nn
import
torch.nn
as
nn
...
@@ -148,7 +148,7 @@ class FeedForward(nn.Module):
...
@@ -148,7 +148,7 @@ class FeedForward(nn.Module):
class
GPTJLayer
(
nn
.
Module
):
class
GPTJLayer
(
nn
.
Module
):
def
__init__
(
self
,
attn
,
ff
,
config
):
def
__init__
(
self
,
attn
,
ff
,
config
):
nn
.
Module
.
__init__
(
self
)
nn
.
Module
.
__init__
(
self
)
self
.
ln_preattn
=
nn
.
LayerNorm
(
config
.
hidden_dim
,
eps
=
config
.
eps
,
device
=
config
.
device
,
dtype
=
config
.
type
)
self
.
ln_preattn
=
nn
.
LayerNorm
(
config
.
hidden_dim
,
eps
=
config
.
eps
,
device
=
config
.
device
,
dtype
=
config
.
d
type
)
self
.
ff
=
ff
(
config
)
self
.
ff
=
ff
(
config
)
self
.
attn
=
attn
(
config
)
self
.
attn
=
attn
(
config
)
self
.
tick
=
True
self
.
tick
=
True
...
@@ -253,12 +253,8 @@ class GPTJConfig:
...
@@ -253,12 +253,8 @@ class GPTJConfig:
eps
:
float
=
1e-5
eps
:
float
=
1e-5
device
:
torch
.
device
=
torch
.
device
(
'cuda'
)
device
:
torch
.
device
=
torch
.
device
(
'cuda'
)
dtype
:
torch
.
dtype
=
torch
.
float16
dtype
:
torch
.
dtype
=
torch
.
float16
Layer
=
GPTJLayer
Layer
:
nn
.
Module
=
GPTJLayer
activation
=
gelu_new
activation
:
Callable
=
gelu_new
def
from_dict
(
self
,
config_dict
):
for
k
,
v
in
config_dict
.
items
():
setattr
(
self
,
k
,
v
)
def
load_gpt_j
(
path
=
"models/6b"
,
state_dict
=
None
):
def
load_gpt_j
(
path
=
"models/6b"
,
state_dict
=
None
):
config
=
{
config
=
{
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment