opendiffusionAI sdxl-longclip (RAW version V0.1)
BUGFIXES!!! (Latest update 2025/05/21)
Please note that the initial release had a bug in the tokenizer config.
tokenizer/tokenizer_config.json
needs to have "model_max_length": 248
Additionally.. I padded out tokenizer_2 and text_encoder_2 so that the normal
python code works with this model now.
What is this?
This is the base SDXL model.. but the CLIP-L text encoder swapped out with "LongCLIP"....
and then the values for CLIP-G zeroed so it has no effect.
In theory, it should be possible to replace the clip-g model with a dummy placeholder, so that the model takes 3G less VRAM overall.
But my attempts at that failed.
Why is this?
SDXL's largest limitations are primarily due to the lousy CLIP(s) used. Not only are they of poor quality, but they have hidden token count
limits, which make effective token count closer to 10.
It is believed that one of the reasons CLIP-G was added on was to work around the limits of original CLIP-L. But.... that makes the model
harder to train, and needlessly takes up more memory and time.
So, I created this version to experimentally prove the better way.
This allows use of up to 248 tokens with SDXL natively, without the layering hacks that some diffusion programs do.
Long Prompt demo
This prompt is stolen fromn the LongCLIP demo prompts:
> The serene lake surface resembled a flawless mirror, reflecting the soft blue sky and the surrounding greenery.
> A gentle breeze played across its expanse, ruffling the surface into delicate ripples that gradually spread out,
> disappearing into the distance. Along the shore, weeping willows swayed gracefully in the light breeze,
> their long branches dipping into the water, creating a soothing sound as they gently brushed against the surface.
> In the midst of this serene scene, a pure white swan floated gracefully on the lake.
> Its elegant neck curved into a graceful arc, giving it an air of dignity.
The point here is that mention of a