Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Sign in / Register
Toggle navigation
S
Stable Diffusion Webui
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Locked Files
Issues
0
Issues
0
List
Boards
Labels
Service Desk
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Security & Compliance
Security & Compliance
Dependency List
License Compliance
Packages
Packages
List
Container Registry
Analytics
Analytics
CI / CD
Code Review
Insights
Issues
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
novelai-storage
Stable Diffusion Webui
Commits
853e21d9
Commit
853e21d9
authored
Oct 18, 2023
by
v0xie
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
faster by using cached R in forward
parent
1c6efdbb
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
14 additions
and
3 deletions
+14
-3
extensions-builtin/Lora/network_oft.py
extensions-builtin/Lora/network_oft.py
+14
-3
No files found.
extensions-builtin/Lora/network_oft.py
View file @
853e21d9
...
...
@@ -57,21 +57,32 @@ class NetworkModuleOFT(network.NetworkModule):
return
R
def
calc_updown
(
self
,
orig_weight
):
# this works
R
=
self
.
R
# this causes major deepfrying i.e. just doesn't work
# R = self.R.to(orig_weight.device, dtype=orig_weight.dtype)
if
orig_weight
.
dim
()
==
4
:
weight
=
torch
.
einsum
(
"oihw, op -> pihw"
,
orig_weight
,
R
)
else
:
weight
=
torch
.
einsum
(
"oi, op -> pi"
,
orig_weight
,
R
)
updown
=
orig_weight
@
R
output_shape
=
[
orig_weight
.
size
(
0
),
R
.
size
(
1
)]
#output_shape = [R.size(0), orig_weight.size(1)]
output_shape
=
self
.
oft_blocks
.
shape
## this works
# updown = orig_weight @ R
# output_shape = [orig_weight.size(0), R.size(1)]
return
self
.
finalize_updown
(
updown
,
orig_weight
,
output_shape
)
def
forward
(
self
,
x
,
y
=
None
):
x
=
self
.
org_forward
(
x
)
if
self
.
multiplier
()
==
0.0
:
return
x
R
=
self
.
get_weight
()
.
to
(
x
.
device
,
dtype
=
x
.
dtype
)
#R = self.get_weight().to(x.device, dtype=x.dtype)
R
=
self
.
R
.
to
(
x
.
device
,
dtype
=
x
.
dtype
)
if
x
.
dim
()
==
4
:
x
=
x
.
permute
(
0
,
2
,
3
,
1
)
x
=
torch
.
matmul
(
x
,
R
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment