Prompt Engineering 101: Generate Sick Stable Diffusion Images

Jeffrey

Nov 18, 2022

8 min read

Anyone can generate sick AI images — you just gotta know how to talk to AI.

If you’ve tried your hand at something like Stable Diffusion and wondered: “How in the hell did they generate that? Why aren’t my prompts working like theirs?!”

Then this is the best place to start.

Even if you’ve generated something spectacular, you may feel like it was dumb luck. The AI feels random. But there’s a method to this madness — we call it prompt engineering.

Give me less than 10 minutes of your life and I’ll teach you how to make something like this:

Or, you can just read the tl;dr

Getting started

To keep things simple, we'll use use Lexica.art. It's a search engine for Stable Diffusion and also allows you to quickly generate SD images right on the site.

Once you're on the website, click on the 'Generate' button.

We'll stick to a couple parameters:

  1. Set your guidance scale to 10.

  2. Set image size to 864x512

These settings can change the results of your images, but we will save this adventure for another time.

Getting started

To keep things simple, we'll use use Lexica.art. It's a search engine for Stable Diffusion and also allows you to quickly generate SD images right on the site.

Once you're on the website, click on the 'Generate' button.

We'll stick to a couple parameters:

  1. Set your guidance scale to 10.

  2. Set image size to 864x512

These settings can change the results of your images, but we will save this adventure for another time.

Getting started

To keep things simple, we'll use use Lexica.art. It's a search engine for Stable Diffusion and also allows you to quickly generate SD images right on the site.

Once you're on the website, click on the 'Generate' button.

We'll stick to a couple parameters:

  1. Set your guidance scale to 10.

  2. Set image size to 864x512

These settings can change the results of your images, but we will save this adventure for another time.

The best way to start is by breaking down the prompt to a good image.

Here’s the prompt from the first image you saw:

illustration of jupiter clouds by dan mumford, alien landscape and vegetation, epic scene, a lot of swirling clouds, high exposure, highly detailed, fantastical, vibrant red tinted colors, uhd

Let’s look at it from the POV of the prompt formula.

illustration (this is my style choice) of

jupiter clouds (the thing I want in the image) +by dan mumford (artist inspo),

alien landscape and vegetation (more stuff I want in the image),

and the rest are all details on how I would like the image to look.

Nice, now you have all that you need to make some amazing images.

It’s time to get you shipping something.

Get wild and put a car on planet Jupiter using Lexica.art. Try using the formula I just showed you.

TL;DR: Prompt Formula

Art Medium + The actual things you want in the image + Art Styles (could be an artist or an actual style like Renaissance) + Modifications (extra details on how the image should look) ⇒ Good Image.

Leveling up your prompt production

I like to imagine that I am sketching on a piece of paper. You add the details step by step to see if it fits the vibes you want.

Here’s my process. Tag along and try it out yourself.

For example, you get this image with just the prompt:

Illustration of Jupiter clouds

Not bad...it might make it on your parent’s fridge.

But what if Dan Mumford drew it?

Illustration of jupiter clouds by dan mumford

Well, here’s an example of Dan’s real art, so it ain’t bad as a start!

What would it look like if a different artist drew it? Let’s try same prompt + different artist.

Illustration of jupiter clouds by aaron horkey

See how it changed the vibes? You can clearly see that the artist’s style makes a big difference.

Let’s add a few more details to our Dan Mumford image so it can have that cool green vibe and a more upscale look.

Heck, let’s add in some more planets too. Yellow on green can look pretty good.

Illustration of jupiter clouds by dan mumford, yellow planets, green tinted colors, uhd

Btw, “uhd” just means ultra-hd lol.

Now this is looking pretty sick and it’s as if we can sketch in Dan Mumford’s style now!

Keep in mind, it took about 3-4 tries to get something I liked — the same prompt will actually give you different results each time.

3-4 times of generating images before changing the prompt is my rule of thumb.

After that, I will start to tweak the prompt to make exactly what I want.

By this point, you are ready to go. You can officially generate images that match your imagination.

All you need to do now is practice. This is the best way to build up insane intuition for how the AI generates images, and how to make it an artistic medium for you.

Now, let’s do something more interesting — let’s study some sick art.

A lil “art theory” for builders

Before we add any additional details to our prompt, it’s good to study the techniques of real artists.

I’m gonna keep this discussion brief though. I want you spending more time making art yourself, rather than reading about it.

The more art you make, the more genius you’ll notice in these artists when you come back to them.

What do you notice when you look at Dan Mumford’s art above?

He uses high exposure. There’s always a strong source of light emitting from the background. His color usages are also rich af.

This gives his art a dramatic look.

How do we capture this with AI?

Let’s go back to the first image you saw here that has the Dan Mumford vibes.

Now check out the prompt with a fresh set of eyes now that you’re a prompt engineering pro.

Illustration of jupiter clouds by dan mumford, alien landscape and vegetation, epic scene, a lot of swirling clouds, high exposure, highly detailed, fantastical, vibrant red tinted colors, uhd

By adding in high exposure + vibrant [any color you want], we are able to capture more of how Dan Mumford would have illustrated the image we want.

You are 99% there

Where does the last 1% come from?

Well, we haven’t even covered the technical aspects like weights, step counter, seed, etc.

But if you’re satisfied with just generating images via prompts, then you’re good to go.

All of the images here were created only through prompts. The images on my Twitter account are also made only through prompts — and there are a lot of wild and beautiful creations there.

You just gotta practice. Mix a few style/artists together and see what you get. Add some styles that an artist typically doesn’t do, like the modifier, fantastical, that I included for my Dan Mumford style.

Let’s make some sick art.

— Jeffrey