On Windows Spine uses a maximum of 1024MB (1GB) of memory by default, which is plenty for most users. You can specify -Xmx1200m when starting Spine, eg:
Spine.exe -Xmx1200m
How much the OS allows Spine to use varies, often 1100-1200MB. You might be able to squeeze up to 1250MB. The memory limitation occurs because for maximum compatibility Spine is 32-bit, though these days that is not so important. In the future we'll move Spine to 64-bit, after which it will be able to use any amount of memory. There are some significant hurdles to doing that though, so given our other plans it's not something we can do in the near future.
The Bleed setting allocates a number of buffers equal to the image size. For large images this would indeed use a lot of memory. We'll look into making this more efficient! Note that using PMA is better: 1) at runtime blending is slightly incorrect when not using PMA, and 2) at runtime when rendering with PMA you can switch between additive and non-additive blending without flushing the batch.
To be sure, you have the Limit memory setting checked?
We have made Spine's texture packer available as free open source in libgdx:
https://github.com/libgdx/libgdx/tree/master/extensions/gdx-tools/src/com/badlogic/gdx/tools/texturepacker
The documentation for the libgdx version is here:
https://github.com/libgdx/libgdx/wiki/Texture-packer
See the part about the nightly build if you just want to get the JAR to run the tool. Running it separately from Spine likely uses less memory, plus you can use 64-bit Java and give Java as much memory as you want. If you run it from a Java class, you can have it run incrementally (it won't pack if the output files are newer than the input files).
The Spine texture packer differs from libgdx only in that Spine's packer can know about a Spine project in order to handle packing meshes with whitespace stripping. If packing meshes with whitespace stripping is important for you, I'm afraid the libgdx packer is not an option. Also in the near future Spine's packer will be able to tightly pack polygons while the libgdx packer cannot.
Edit: I've improved Bleed to use less memory in 3.7.92. :nerd:
