Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Sign in / Register
Toggle navigation
S
Stable Diffusion Webui
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Locked Files
Issues
0
Issues
0
List
Boards
Labels
Service Desk
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Security & Compliance
Security & Compliance
Dependency List
License Compliance
Packages
Packages
List
Container Registry
Analytics
Analytics
CI / CD
Code Review
Insights
Issues
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
novelai-storage
Stable Diffusion Webui
Commits
d6d0b22e
Commit
d6d0b22e
authored
Nov 15, 2023
by
v0xie
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fix: ignore calc_scale() for COFT which has very small alpha
parent
7edd50f3
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
11 deletions
+5
-11
extensions-builtin/Lora/network_oft.py
extensions-builtin/Lora/network_oft.py
+5
-11
No files found.
extensions-builtin/Lora/network_oft.py
View file @
d6d0b22e
...
...
@@ -99,12 +99,9 @@ class NetworkModuleOFT(network.NetworkModule):
is_other_linear
=
type
(
self
.
sd_module
)
in
[
torch
.
nn
.
MultiheadAttention
]
if
not
is_other_linear
:
#if is_other_linear and orig_weight.shape[0] != orig_weight.shape[1]:
# orig_weight=orig_weight.permute(1, 0)
oft_blocks
=
self
.
oft_blocks
.
to
(
orig_weight
.
device
,
dtype
=
orig_weight
.
dtype
)
#
without this line the results are significantly worse / less accurate
#
ensure skew-symmetric matrix
oft_blocks
=
oft_blocks
-
oft_blocks
.
transpose
(
1
,
2
)
R
=
oft_blocks
.
to
(
orig_weight
.
device
,
dtype
=
orig_weight
.
dtype
)
...
...
@@ -118,9 +115,6 @@ class NetworkModuleOFT(network.NetworkModule):
)
merged_weight
=
rearrange
(
merged_weight
,
'k m ... -> (k m) ...'
)
#if is_other_linear and orig_weight.shape[0] != orig_weight.shape[1]:
# orig_weight=orig_weight.permute(1, 0)
updown
=
merged_weight
.
to
(
orig_weight
.
device
,
dtype
=
orig_weight
.
dtype
)
-
orig_weight
output_shape
=
orig_weight
.
shape
else
:
...
...
@@ -132,10 +126,10 @@ class NetworkModuleOFT(network.NetworkModule):
return
self
.
finalize_updown
(
updown
,
orig_weight
,
output_shape
)
def
calc_updown
(
self
,
orig_weight
):
multiplier
=
self
.
multiplier
()
*
self
.
calc_scale
()
#
if self.is_kohya:
# return self.calc_updown_kohya(orig_weight, multiplier
)
#else:
# if alpha is a very small number as in coft, calc_scale will return a almost zero number so we ignore it
#
multiplier = self.multiplier() * self.calc_scale()
multiplier
=
self
.
multiplier
(
)
return
self
.
calc_updown_kb
(
orig_weight
,
multiplier
)
# override to remove the multiplier/scale factor; it's already multiplied in get_weight
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment