:: Home :: Get Hosted :: PPM FAQ :: Forum FAQ :: Search :: Memberlist :: Usergroups :: Register :: Profile :: Log in to check your private messages :: Log in ::
Do you want to advertise at Project Perfect Mod. Find out how to do it HERE.

The time now is Mon Dec 18, 2017 1:37 am
All times are UTC + 0
 Forum index » Featured Tools » CnC Editing Tools
Image Shaper
Moderators: CnC Editing Tools Developpers
Post new topic   Reply to topic Page 1 of 2 [56 Posts] View previous topic :: View next topic
Goto page: 1, 2 Next
Author Message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Sun Dec 03, 2017 10:25 pm    Post subject:   Image Shaper
Subject description: a tool to convert image sequences into SHPs
Reply with quote

This program allows to combine images and convert the sequence of images into a SHP file.
It's basically the more powerful successor of Image Combine

Download: ImageShaper.zip Version 01.01.00.06
Installation:
-extract the zip into an empty folder with read and write permission
-start the ImageShaper.exe


This tool was created to overcome the flaws of SHP Builder, which are
-very slow import (UI freezes during import)
-impossible to combine different render passes during import
-inconvenient palette management, which needs a custom palette for each import that wants certain colors to be ignored (e.g. glowing colors, non-remap reds etc)
-impossible to import already paletted images. SHP Builder still applies it's own color conversion, which messes up different palette colors with the same RGB value (e.g. a glowing pixel with color #240 is changed into non-glowing color #15, non-remap reds changed into remap).


With this tool you can convert up to 3 different render passes in one operation into a single SHP, while using only a single palette file.
This palette gets assigned to each image with possibly different settings, like ignore glowing colors.


For those who like comparison numbers:
import of 576 files with 200x200 pixel size
SHP Builder: over 3 minutes
Image Shaper: 25 seconds

How to use
Drag and Drop files on the datagrid or use its right-click context menu to load image files.


Assign a palette to the images.

-right click on the palette and load a palette


-right click on a color which should be ignored during the conversion into the palette


-the color is marked red

-hold-click on that color and drag the mouse to other colors, which should have the same "ignored" property


-select all images, right-click on the selected images and set "Set palette", to assign this palette with these modifications to the selected files


-clicking again on a single image, the preview will show it already in the final converted state


-drag and drop (or load) a second set of images in the 2nd column, which should be combined with the respective image in the first column


-select the images in the 2nd column, do the necessary changes to the palette

-in this case, dark pixel should be converted into transparent. Thus "Make Transparent" is used instead of "Ignore Color", as the last one would force dark pixel to be converted into the only left colors, which are the glowing colors


-after the palette changes are done, apply the palette to the selected files


-this way we got rid of dark anti-aliased pixel and have only the glowing pixel left in the right palette color


-in this example another set of a images of glowing pixel from a non-anti-aliased render pass is applied to the 3rd column
-these got again a certain modified palette assigned


With the non-antialiased glowing pixel...

...the anti-aliased glowing shapes...

...and the unit without glowing pixel, ...

...a click on the row to the left, shows the combined images in the preview.


Finally a click on [Start], will start the conversion of all the files into a single SHP


a few final adjustments can then be done in SHP Builder



Version 01.00.00.00
-first public release

Version 01.00.00.02
-(bugfix) compiled as x86 to match the x86 MagickLibrary and prevent mismatch on x64 systems

Version 01.01.00.00
-(update) SHP creating routine implemented, which allowed to get rid of the big ImageMagick library and the limited ShapeSet tool.
-(update) SHP reading routine implemented, which allows to load SHPs and have the frames listed for further resaving and modification.
loaded SHPs are not color converted again and read/shown as indexed palette image. Color Conversion methods are not applied on these.
-(update) Radar color setting added, allowing to modify/set each frames radar color. Used by TS/RA2 only for Overlay SHPs like tiberium/ore.
-(update) Added option to create an image for each processed frame.
-(update) Added option to prevent wobble bug in TS. This fixes turreted SHP units and prevents them from wobbling 1 pixel up down when rotating the turret or when using a moving animation.
-(update) Options to set the compression for each frame.
-(update) Hovering the mouse over palette indexed pixel in the preview image, highlights the used color of that pixel in the palette

Version 01.01.00.01
-compiled as target platform "any CPU"

Version 01.01.00.02
-focus is set back to the datagrid after SHP conversion process is finished
-SHPs with wrong compression bit set are loaded anyway
-last used settings stored/loaded from ImageShaper.ini

Version 01.01.00.03
-(bugfix) bad ini values for position and size don't crash anymore
-(update) simple palette manager added, which keeps track of different palette settings
-(update) image files only point to a palette from the manager and don't keep a full copy of it, thus hopefully reducing memory usage
-(update) palette edits are instantly applied to all files sharing the currently selected palette
-(update) performance of deleting files/frames improved

Version 01.01.00.04
-(update) images/frames store only an int index to the used palette from the palettemanager, thus reducing memory usage per image/frame

Version 01.01.00.05
-(bugfix) palettemanager crashed on images with unassigned palettes

Version 01.01.00.06
-a few adjustments on the class for the internal stored images/frames
-progressbar is showing the current progress of the imported files

P.S.: i did some extensive testing on different color conversion methods, and the default "Euclidean", which gives similar results to SHP Builders "3D RGB Structuralis", is actually better than DeltaCIE2000.
DeltaCIE2000 has big problems with darker shades and should be used only on brighter colors.
So the SHP Builder color conversion is actually not that bad as mentioned here, it's just slow.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Last edited by Lin Kuei Ominae on Thu Dec 14, 2017 2:46 pm; edited 12 times in total

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Sun Dec 03, 2017 11:14 pm    Post subject: Reply with quote

Great idea piping the output into the shapeset tool. Not open sourcing it though?

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Mon Dec 04, 2017 6:49 am    Post subject: Reply with quote

Error message when pressing start:
Could not load file or assembly 'Magick.NET-Q8-x86, Version=7.1.0.0, Culture=neutral, PublicKeyToken=2004825badfa91ec' or one of its dependencies. An attempt was made to load a program with an incorrect format.

The mentioned DLL exist in the folder and started the program by double
clicking the EXE. The source files were PNG files exported from SHP Builder.

For the problem you mentioned on paletted image or retaining the exact
color, I have been using SHP Builder r96 as the last good SHP Builder.
Later versions behave differently.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Mon Dec 04, 2017 9:39 am    Post subject: Reply with quote

fixed
the project was compiled as any CPU while the MagickLibrary is x86 only, thus it wasn't working on x64 machines.
It's now compiled as x86 to always match.

Download again from first post to get version 02


Blade wrote:
Not open sourcing it though?

I can send the source code to those who want it, but usually i don't make my tools open source.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Mon Dec 04, 2017 10:27 am    Post subject: Reply with quote

Now DLL loads but says in #conversionlog:
processing... *** ERROR *** : Too many filenames specified.

It creates the pcx files but not shp, though it says created shp.
752 files used for test.

Edit: Space in the source filenames causes it.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Mon Dec 04, 2017 10:50 am    Post subject: Reply with quote

shit, ShapeSet has it's limit there.
The message "shp created" simply comes when the program notices the separate ShapeSet process has finished working. I didn't check for errors from ShapeSet.

Ok, i guess this means we can scrap this tool, since ShapeSet is too limited. Fuck. Wall bash!


Anyone interested in writing a simple .NET routine to create an SHP from a set of indexed palette images? I really have no interest in trying to decipher the SHP format.
Best would be something like this
Code:
CSHPFile shp=new CSHPFile(filename);
shp.AddFrame(Bitmap image);
shp.Save();

Then i could simply pump the generated frames into the class, without having to use the old PCX file format or any temporary files at all.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
PussyPus
Energy Commando


Joined: 14 Jul 2015
Location: The Arab Country, Time Travel?

PostPosted: Mon Dec 04, 2017 11:01 am    Post subject: Reply with quote

Now that would be easier to make SHPs
_________________
If you are a MetalHead (Heavy Metal Fan) and don't want to be a metalhead, Just remove your metal ball from your head. �:p .


Back to top
View user's profile Send private message Skype Account
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Mon Dec 04, 2017 11:06 am    Post subject: Reply with quote

SHP Builder export had space in the filenames, when that space was
removed, it succeeded for 752 frames but not quad of it at 10528 frames.

Back to top
View user's profile Send private message
Graion Dilach
Defense Minister


Joined: 22 Nov 2010
Location: Iszkaszentgyörgy, Hungary

PostPosted: Mon Dec 04, 2017 11:07 am    Post subject: Reply with quote

Imma just leaving these two links here. Tool looks really promising, but not gonna try writing a class for code I don't even know.

https://github.com/OpenRA/OpenRA/blob/bleed/OpenRA.Mods.Common/UtilityCommands/ConvertPngToShpCommand.cs
https://github.com/OpenRA/OpenRA/blob/bleed/OpenRA.Mods.Common/SpriteLoaders/ShpTSLoader.cs
_________________



AS Discord server: https://discord.gg/7aM7Hm2

Back to top
View user's profile Send private message Skype Account
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Mon Dec 04, 2017 11:37 am    Post subject: Reply with quote

E1 Elite wrote:
it succeeded for 752 frames but not quad of it at 10528 frames.

ShapeSet's limit is at 1019 frames.

Thanks GD. Maybe i'll find some time and interest in near future to find out how to code a simple SHP creating routine. Then these links surely come in handy.

\edit
GD, is there no writer for TS SHPs?
I only found ShpTDSprite.Write
https://github.com/OpenRA/OpenRA/blob/bleed/OpenRA.Mods.Common/SpriteLoaders/ShpTDLoader.cs

but there is no ShpTSSprite.Write
https://github.com/OpenRA/OpenRA/blob/bleed/OpenRA.Mods.Common/SpriteLoaders/ShpTSLoader.cs
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
Graion Dilach
Defense Minister


Joined: 22 Nov 2010
Location: Iszkaszentgyörgy, Hungary

PostPosted: Tue Dec 05, 2017 2:38 pm    Post subject: Reply with quote

Lol, well, seems so.

TBH, never used that command myself - even on Linux, I just Wine SHP Builder and/or Mixer instead of dealing with ORA utility.

Heard from Nolt however that the tool keeps the SHP frames as pcx as well - that's cool already.
_________________



AS Discord server: https://discord.gg/7aM7Hm2

Back to top
View user's profile Send private message Skype Account
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Tue Dec 05, 2017 4:06 pm    Post subject: Reply with quote

yes, palette indexed PCX are stored in the Temp subfolder, as the first step before ShapeSet converts these to SHP.

I would have preferred a more modern fileformat, to get rid of the huge ImageMagick library just for that old format, but ShapeSet seems to accept PCX only.
Is OpenRA supporting PCX files directly as frames? Does Nolt use PCX as well or another fileformat?
Would be surely useful if in future the tool keeps the option to create single palette indexed frame images in the OpenRA supported image format.


I've just started extracting only the relevant code from OpenRA to understand the TS SHP format.
Currently i can load and view already an SHP (huge thanks to the OpenRA devs for that )
and just started writing a SHP writer routine.

Once i can write (TS)SHP files directly, i can get rid of ImageMagick, ShapeSet and create the SHP directly with hopefully no frame limit (except the inherent limit of SHPs, supporting only an ushort number of frames 65535).



P.S. Did i noticed right that OpenRA ignores the radar color of SHPs?
There are the 3 byte values 4 byte after the compression format byte, but the OpenRA reader simply jumps over them
in class ShpTSFrame : ISpriteFrame
Code:

Format = StreamExts.ReadUInt8(s);
s.Position += 11;
FileOffset = StreamExts.ReadUInt32(s);

I'm not 100% sure, but i think
s.Position+=4;
is then followed by the 3 RGB byte values. (IIRC used on tiberium SHPs to hold the radarcolor of the tiberium patch)
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
tomsons26lv
Scorpion Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Tue Dec 05, 2017 9:16 pm    Post subject: Reply with quote

Don't use ORA's code for LCW/XORDelta, they never fixed it even after we pointed out the issues with it.
Nyer has been playing around with our code and making a viewer for initially various WW files, now extended to support other similarly built files.
http://nyerguds.arsaneus-design.com/project_stuff/2017/CnC64FileConverter/dev/CnC64FileConverter_v1.2.7_src.rar
Code in it is practically our code converted to C#.
\Domain\GameData\Westwood\WWCompression.cs
He doesn't have reading TS SHP's yet but only compression that has is RLE, i wouldn't be too surprised that WW's existing RLE implementations are it already.

The colors you are referring to are not radar colors, its the average color of all the non-keyed(0x0 index) pixels in the frame, it might be used as radar color indeed, maybe some other uses, we've yet to find the exact code that does it.
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Wed Dec 06, 2017 8:00 am    Post subject: Reply with quote

If i understand it right, LCW/XORDelta is only used for SHP(TD) files, not on SHP(TS) files.
So i can ignore this, since i don't care about TD SHPs. If anyone wants them he can open the generated SHP(TS) and save it as SHP(TD) again using SHP Builder.

RLE_Encoding algorithm could be indeed useful, as this is from what i can see used in SHP(TS) format.

Since i use OpenRA's code to read/show SHPs and it works fine on SHPs generated with the official Westwood ShapeSet tool, i assume that the implementation of OpenRA is correct.
Gonna have to test a few more example though, also some crazy coded SHPs with unusual frames.


Thanks for the source. Also thanks to Nyerguds Approved!
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Thu Dec 07, 2017 10:38 am    Post subject: Reply with quote

ok, i can create uncompressed SHP(TS) files. yay
SHP Builder loads them fine and they are basically exactly the same as the SHPs produced by ShapeSet, if you don't use the -R parameter.

But,
SHP Builder has different compression methods.
Simply sending my image data byte-array into Nyerguds "RleEncode" routine and then writing this, ends in total garbage when viewed in SHP Builder. I've also set the format byte-flag to 3 and the headers look the same.

So, does anyone know how to use the RleEncode routine?
It has for example a bool "swapWords" parameter, though i've no clue if i have to set it to true or false.
"minimumRepeating" is another parameter, where i've no clue if i should set it to 2, 3 or any other value. (tried 2 and 3, but the result is nothing alike the ones produced by ShapeSet with -R parameter and/or SHP Builder)


I almost have the impression the compression is done only on each line, not the entire byte-array of the image, because the hex code of the SHP builder/ShapeSet generated files have some distinctive line patterns left, while Nyerguds RleEncoded bytes are just a block of completely random bytes.


\Edit
after some byte analysis, i think SHP Builder and ShapeSet are compressing zero bytes only, while Nyerguds RleEncoder seems to compress all doubled pixel, regardless of their color.
Thus his routine is not suitable for me. Sad
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Thu Dec 07, 2017 1:42 pm    Post subject: Reply with quote

Why don't you try zzattack's map renderer which should be in C#, and should
lead you to quick result.

Nyerguds code looks generic and it may not be specifically ready to use as
such for SHP.

There are plenty of source code - stucuk's SHP Toolz, XCC or SHPBuilder
or OpenRA stuff or who else has it.

Back to top
View user's profile Send private message
Banshee
Supreme Banshee


Joined: 15 Aug 2002
Location: Brazil

PostPosted: Thu Dec 07, 2017 2:10 pm    Post subject: Reply with quote

SHP Builder's RLE routine, based on XCC Mixer, only compress color #0, LKO.

Back to top
View user's profile Send private message Visit poster's website Skype Account
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Thu Dec 07, 2017 2:30 pm    Post subject: Reply with quote

E1 Elite wrote:
Why don't you try zzattack's map renderer which should be in C#, and should
lead you to quick result.

didn't thought about that, but i think it most likely has only loading, not writing.
\Edit
yes, only loading.

E1 Elite wrote:

Nyerguds code looks generic and it may not be specifically ready to use as
such for SHP.

There are plenty of source code - stucuk's SHP Toolz, XCC or SHPBuilder
or OpenRA stuff or who else has it.

SHPBuilder is a pain to read, but i'm currently trying to understand and convert its code.
OpenRA has only loading, not writing SHP(TS) format.
I can find on XCC only infos about SHP(TD), not SHP(TS) format
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
tomsons26lv
Scorpion Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Fri Dec 08, 2017 12:06 am    Post subject: Reply with quote

Here's some pseudo if it helps
encoder from ShapeSet
Code:
int __stdcall RLE_40D530(char *source, _BYTE *a2, int length)
{
    int v3; // ST60_4@16
    signed int v5; // [sp+4Ch] [bp-18h]@13
    int i; // [sp+50h] [bp-14h]@8
    char *v7; // [sp+54h] [bp-10h]@5
    int v9; // [sp+5Ch] [bp-8h]@5

    if ( !source )
    {
        _assert(
            "source != NULL",
            "c:\\ra2\\shapeset\\rle.cpp",
            dword_441E30 + 1);
    }
    if ( length <= 0 )
    {
        _assert("length > 0", "c:\\ra2\\shapeset\\rle.cpp", dword_441E30 + 2);
    }
    v9 = 0;
    v7 = a2;
    while ( length > 0 )
    {
        if ( *source )
        {
            if ( v7 )
            {
                *v7++ = *source++;
            }
            else
            {
                ++source;
            }
            ++v9;
            --length;
        }
        else
        {
            for ( i = 0; !source[i] && i <= length; ++i )
            {
                ;
            }
            if ( i < 0xFF )
            {
                v5 = i;
            }
            else
            {
                v5 = 0xFF;
            }
            if ( v7 )
            {
                *v7 = 0;
                v3 = (v7 + 1);
                *v3 = v5;
                v7 = (v3 + 1);
            }
            v9 += 2;
            source += v5;
            length -= v5;
        }
    }
    return v9;
}

Decoder? from ShapeSet
Code:
int __stdcall RLE_40D7C0(char *source, _BYTE *dest, int length)
{
    char *v4; // [sp+4Ch] [bp-14h]@10
    char v5; // [sp+50h] [bp-10h]@9
    _BYTE *v7; // [sp+58h] [bp-8h]@7

    if ( !source )
    {
        _assert(
            "source != NULL",
            "c:\\ra2\\shapeset\\rle.cpp",
            dword_441E38 + 1);
    }
    if ( !dest )
    {
        _assert("dest != NULL", "c:\\ra2\\shapeset\\rle.cpp", dword_441E38 + 2);
    }
    if ( length <= 0 )
    {
        _assert("length > 0", "c:\\ra2\\shapeset\\rle.cpp", dword_441E38 + 3);
    }
    v7 = dest;
    while ( length > 0 )
    {
        v5 = *source++;
        --length;
        if ( v5 )
        {
            *v7++ = v5;
        }
        else
        {
            v4 = *source++;
            --length;
            while ( v4 > 0 )
            {
                *v7++ = 0;
                --v4;
            }
        }
    }
    return v7 - dest;
}
[/code]
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Sun Dec 10, 2017 12:55 pm    Post subject: Reply with quote

new version released
this one has seen a big amount of changes, mainly direct SHP reading and writing support. So I was able to get rid of ImageMagick and ShapeSet.
Probably the best change however is the option to prevent the annoying TS wobble bug.

Version 01.01.00.00
-(update) SHP creating routine implemented, which allowed to get rid of the big ImageMagick library and the limited ShapeSet tool.
-(update) SHP reading routine implemented, which allows to load SHPs and have the frames listed for further resaving and modification.
loaded SHPs are not color converted again and read/shown as indexed palette image. Color Conversion methods are not applied on these.
-(update) Radar color setting added, allowing to modify/set each frames radar color. Used by TS/RA2 only for Overlay SHPs like tiberium/ore.
-(update) Added option to create an image for each processed frame.
-(update) Added option to prevent wobble bug in TS. This fixes turreted SHP units and prevents them from wobbling 1 pixel up down when rotating the turret or when using a moving animation.
-(update) Options to set the compression for each frame.
-(update) Hovering the mouse over palette indexed pixel in the preview image, highlights the used color of that pixel in the palette



@E1 Elite: i successfully created a 48MB SHP with over 40.000 frames. Wink
So feel free to test out the limits of the SHP format and this tool. Smile

Thanks again to the devs of OpenRA and OS SHP Builder, who helped creating this.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Sun Dec 10, 2017 6:05 pm    Post subject: Reply with quote

Currently the EXE v1.01.00.00 is not able to handle more than 2gb memory,
it gives out of memory exception.

I modified the EXE to set that flag - 0x96 from 02 to 22 to fix that.

It is able to make SHP upto 22K frames for a dimension I used but with smaller
file dimensions it was able to do more.

The drag and drop dialog box doesn't show more than a specified number of
files based on its size. More files with preview box open, gives exception:

System.ArgumentException: Parameter is not valid.
  at System.Drawing.Bitmap..ctor(Int32 width, Int32 height, PixelFormat format)
  at System.Drawing.Bitmap..ctor(Int32 width, Int32 height)
  at ImageShaper.UC_ImageCanvas.UpdateZoomedPicture()
  at ImageShaper.UC_ImageCanvas.SetImage(Image image)
  at ImageShaper.Form_DockPreview.SetImage(Bitmap bitmap)
  at ImageShaper.Form_ImageShaper.<UpdatePreview>b__a(Object w_s, ProgressChangedEventArgs w_e)
  at System.ComponentModel.BackgroundWorker.OnProgressChanged(ProgressChangedEventArgs e)
  at System.ComponentModel.BackgroundWorker.ProgressReporter(Object arg)

A command line utility could be an option which would avoid the GUI
limitations.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Sun Dec 10, 2017 6:18 pm    Post subject: Reply with quote

E1 Elite wrote:
Currently the EXE v1.01.00.00 is not able to handle more than 2gb memory,
it gives out of memory exception.

I modified the EXE to set that flag - 0x96 from 02 to 22 to fix that.

is that like changing the x86 platform target to x64?

What kind of gigantic SHP are you trying to create? o.O

E1 Elite wrote:
The drag and drop dialog box doesn't show more than a specified number of
files based on its size. More files with preview box open, gives exception:

System.ArgumentException: Parameter is not valid.
  at System.Drawing.Bitmap..ctor(Int32 width, Int32 height, PixelFormat format)
  at System.Drawing.Bitmap..ctor(Int32 width, Int32 height)
  at ImageShaper.UC_ImageCanvas.UpdateZoomedPicture()
  at ImageShaper.UC_ImageCanvas.SetImage(Image image)
  at ImageShaper.Form_DockPreview.SetImage(Bitmap bitmap)
  at ImageShaper.Form_ImageShaper.<UpdatePreview>b__a(Object w_s, ProgressChangedEventArgs w_e)
  at System.ComponentModel.BackgroundWorker.OnProgressChanged(ProgressChangedEventArgs e)
  at System.ComponentModel.BackgroundWorker.ProgressReporter(Object arg)

A command line utility could be an option which would avoid the GUI
limitations.

what do you mean with drag&drop dialog box? D&D has no dialog. Do you mean the context menu "Load Images" option?

I don't know how that has anything to do with the preview window, since that shows only a single image, so it shouldn't matter how many files you have in the list nor how much you currently load.  Confused

Can you give me the file that throws the exception?
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Sun Dec 10, 2017 6:38 pm    Post subject: Reply with quote

x86/x64 is for which compiler to use. Greater than 2gb is a EXE file header
characteristics flag which allows it to handle more than 2 gb address space.

Drag and drop is same as the load images to the list.

I exported Orac's heavy mech fixed orange_w42.shp and replicated to 64k.
https://ppmforums.com/viewtopic.php?p=528764#528764

Edit: Can you provide both x86 and x64 EXEs?

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Sun Dec 10, 2017 9:26 pm    Post subject: Reply with quote

i can only compile as target platform "any CPU" on my 32 bit PC.
download again and test version 1.1.0.1 if that works better.


Though i think such unusual huge SHP aren't common or used in the first place.
I doubt the games would even handle/load a SHP that takes up several Gigabyte.

In my tests, 65535 frames of 100x100 pixel loaded fine in the list and also got converted into a SHP.

For the usual or even extraordinary unit that isn't reaching more than 5000 frames max for 32 facings even when having 30 walk, 30 firing, 30 death and 30 standing frames, the current version should work well enough.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Sun Dec 10, 2017 10:27 pm    Post subject: Reply with quote

Lin Kuei Ominae wrote:
fixed
Blade wrote:
Not open sourcing it though?

I can send the source code to those who want it, but usually i don't make my tools open source.


Any particular reason? The code may be hacky and not particularly pretty, but there is no commercial value to you in keeping it closed source and it gives others a chance to continue it if you drop out of the scene. Take the vqa encoder Ugordan made for example, no one can improve or port that since the source was never released for any version of it.

Also, if you have started using code from OpenRA or XCC, then barring permission from whoever wrote it, I believe the code is all under the GPL version 3 so you would be obliged to release the source for anything you wrote based on it if you wanted to comply with the license terms.

Basically there is little to no benefit to you in keeping it closed source and you may be obliged to do it anyway by other code you've incorporated. Just throw it up on github and call it done.

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Mon Dec 11, 2017 6:14 am    Post subject: Reply with quote

Lin Kuei Ominae wrote:
i can only compile as target platform "any CPU" on my 32 bit PC.
download again and test version 1.1.0.1 if that works better.

Nice, it runs as 64-bit app in x64 system and also runs in x86 system. The
loading images is a bit faster.

Was able to make SHP out of 65534 frames and viewed it in SHP Builder.

Adding to the list (load images), it is still not able to open all the 64K files
in one step if I load the palette first and then load images. First it loaded
32760 then added again twice to get the rest.

Yet to test in-game.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Mon Dec 11, 2017 5:43 pm    Post subject: Reply with quote

@Blade
attached the source for the SHP reading and creating routines that i used.
Copied and converted together from OpenRA and SHP Builder.
The classes encapsulated and modified for easier usage in this project.
It's basically only
CShaPer.CreateSHP(...)//bool keepEvenFrameSizes is used to set "prevent TS wobble bug"
CShaPer.GetFrames() //reads the header only, not the entire SHP image data (the OpenRA methods adjusted a bit for this)
CShaPer.GetFrame() //returns the bitmap of a single frame


The other classes are too much intertwined with all my other tools, and i have no interest to extract the parts that are used on this tool only, nor to make all tools (including non C&C related projects) public.


CSHaPer.zip
 Description:

Download
 Filename:  CSHaPer.zip
 Filesize:  10.16 KB
 Downloaded:  2 Time(s)


_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Mon Dec 11, 2017 8:32 pm    Post subject: Reply with quote

I wasn't trying to force your hand or anything, most people won't even be interested in the code in this community and its not like the OpenRA team or even Olaf are really likely to pursue the matter and they are the only ones who would have standing to do anything about enforcing the GPL on you. As it is, if I can't rebuild the entire tool, a bit of the source isn't exactly useful.

You should be careful about incorporating open source code in your tools if you don't intend to play by the open source rules though, the GPL requires you make the entire source available for a binary you distribute that contains some GPL code, not just the bit of GPL code you happen to use.

I personally think its just the right thing to do to open source tools for modding. I am aware that I have released tools myself without source because they were based on code I couldn't just okay the release of on my own, but if I could have I would.

Back to top
View user's profile Send private message
Bittah Commander
Defense Minister


Joined: 21 May 2003
Location: The Netherlands

PostPosted: Mon Dec 11, 2017 10:37 pm    Post subject: Reply with quote

After clicking start and the operation finishes, the cursor jumps to the status panel, which means that you can't immediately press delete to remove all frames and this slows down your work pace when you have a multiple files to process.

Also, for some reason I can't import the attached SHPs into the tool (others worked fine).


mwavemsam.shp
 Description:

Download
 Filename:  mwavemsam.shp
 Filesize:  37.56 KB
 Downloaded:  1 Time(s)


LTNKCRUS.shp
 Description:

Download
 Filename:  LTNKCRUS.shp
 Filesize:  21.2 KB
 Downloaded:  1 Time(s)


_________________

Back to top
View user's profile Send private message Send e-mail YouTube User URL Facebook Profile URL
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Tue Dec 12, 2017 3:45 pm    Post subject: Reply with quote

The OpenRA function to determine if they are a valid SHP returns false.
mwavemsam has 0 as compression set on all frames, valid are only 1 and 3 for SHP(TS)
same is true for ltnkcrus

Whatever program was used to create these, that program sucks at writing correct SHP files.

I'll see if i can adjust the routine to verify SHP format files to allow such bad written files.


\Edit
try version 02
it should load these bad SHPs

Version 01.01.00.02
-focus is set back to the datagrid after SHP conversion process is finished
-SHPs with wrong compression bit set are loaded anyway
-last used settings stored/loaded from ImageShaper.ini
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
Bittah Commander
Defense Minister


Joined: 21 May 2003
Location: The Netherlands

PostPosted: Tue Dec 12, 2017 10:26 pm    Post subject: Reply with quote

I believe I just used OS SHP Builder to create those SHPs, but there's a (small) chance that I used XCC Mixer.

Edit: apparently ImageShaper crashes whenever it's minimized by whatever means and it'll be invisible when you start it back up afterwards because it's apparently way off-screen. For some reason ImageShaper.ini then has this:
Code:
[General]
X_StartPosition=-32000
Y_StartPosition=-32000

_________________

Back to top
View user's profile Send private message Send e-mail YouTube User URL Facebook Profile URL
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Wed Dec 13, 2017 5:02 am    Post subject: Reply with quote

Option for usage of more than 2gb memory:

Go to <VS Install Folder>\VC\bin\

Use command like

editbin.exe /LARGEADDRESSAWARE ImageShaper.exe

It sets the same flag that I mentioned earlier. Should benefit x86 users.

When a palette is loaded and then loading images, it assigns the palette
to the loading images which was hitting the 4gb limit, that was the reason
why it was not loading all 64k images at once. It doesn't increase memory
consumption on loading images when no palette is loaded and the program
is maximized, so loads all 64k images in one go.

On right click, reset palette or clear palette would be helpful. Also on the
left panel right click, clear all images option could be added.

Possibly a check or warning could be added if some image has not yet
set with a palette before running the start process.

A select all for one column option would be helpful if already some hotkey
is not available. Ctrl+A selects all files on all columns.

After resizing and probably when in background it crashes and sets INI values as
X_StartPosition=-32000
Y_StartPosition=-32000
Then INI needs to be deleted or those values set to 0.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Wed Dec 13, 2017 2:47 pm    Post subject: Reply with quote

done, new version uploaded

Version 01.01.00.03
-(bugfix) bad ini values for position and size don't crash anymore
-(update) simple palette manager added, which keeps track of different palette settings
-(update) image files only point to a palette from the manager and don't keep a full copy of it, thus hopefully reducing memory usage
-(update) palette edits are instantly applied to all files sharing the currently selected palette
-(update) performance of deleting files/frames improved

@E1: i've deliberately not yet set /LARGEADDRESSAWARE, due to the changed behaviour with the new palette manager.
With this, each added file should in theory only have a pointer to a very small list of different palette setups, with files/frames having the same palette setup also pointing to the same palette object. Thus the memory used for each each file should be very small and not a full copy of an entire palette setup for each frame.
Would be nice if you could test this and verify if true.


With the new palette manager, you also don't have to do "Set Palette" to the images/frames after each color modification, since the palette you see is the same object that the image/frame is using.
This should also speed up the workflow to set up the different files.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Wed Dec 13, 2017 3:44 pm    Post subject: Reply with quote

You could always apply the >2gb patch, it only makes a program capable
of using 3gb on x86.

The case of loading images with a palette loaded is still the same. It reaches
the 4gb limit and doesn't load all files in one go. But the improvement is
now we can unload the palette and then load images and it loads all images
without needing the program to be maximized.

The palette manager dropdown should show full path of the palette as a
same named palette could have been loaded from some other folder.

Select all in a column works fine which replaces earlier sort function.

If you want to increase complexity, it could be made shadow frames aware
like SHP Builder.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Wed Dec 13, 2017 4:11 pm    Post subject: Reply with quote

Ok, i'll see first if i can improve that in a more elegant way. Making it largeaddressaware seems to be a pretty cheap solution imo.
E1 Elite wrote:
If you want to increase complexity, it could be made shadow frames aware
like SHP Builder.

hmm, a simple "take second half as shadow" seems very imprecise imo (e.g. animations don't have them).
The user also has to give the shadow frames a different palette setup anyway and i have no idea how to make the right palette setup automatically, since each Modder may use different background and shadow colors when rendering stuff.

Were there any tests done showing that shadow frames need a different/certain compression to work fine?
Otherwise there is also no need to differentiate normal from shadow imo.


I have my C&C tools in a rather long subfolder
d:\Modding\Tiberian Sun\ImageShaper\
which would make the name of a palette very long and in most cases the filename itself unreadable in the short combobox. Thus i left only the filename + numbered suffix for different versions.
I wouldn't like to make the palette combobox super wide just for the path, which is usually the subfolder \Palettes
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Wed Dec 13, 2017 4:22 pm    Post subject: Reply with quote

Most would be using the same palette for both normal and shadow frames.
Only in case like RA2 civ buildings where a different palette is used for rubble
frames, you could automatically assign the same palette for its shadow frame.

Not sure what tests were done but SHP Builder has an option for having
best compression or RLE_0 for normal frames and none for shadow frames.

What about pcx file format support?

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Wed Dec 13, 2017 4:46 pm    Post subject: Reply with quote

.NET has no PCX support and i don't add the huge ImageMagick library for this outdated format again.

new version uploaded
please test again
Version 01.01.00.04
-(update) images/frames store only an int index to the used palette from the palettemanager, thus reducing memory usage per image/frame

\Edit
Version 01.01.00.05
-(bugfix) palettemanager crashed on images with unassigned palettes
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Wed Dec 13, 2017 5:28 pm    Post subject: Reply with quote

v1.01.00.05

Loading images with palette loaded case remains the same reaching 4gb limit.

Could try it in 2 passes, 1st as if no palette is loaded and 2nd assign the loaded
palette. In the meantime could show busy cursor or not show the list or show
progress bar etc. so that user knows when it has finished loading.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Wed Dec 13, 2017 6:20 pm    Post subject: Reply with quote

No clue how palette or no palette can still make a difference. Confused
Each cell in the DataGrid stores only a single integer now for the palette.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
tomsons26lv
Scorpion Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Wed Dec 13, 2017 8:29 pm    Post subject: Reply with quote

E1 Elite wrote:

Not sure what tests were done but SHP Builder has an option for having
best compression or RLE_0 for normal frames and none for shadow frames.

What about pcx file format support?

Its RLE, RLE compresses same bytes to one byte and repeat count, "aaaabbcdeeeeefghhhij" for example would become "a4b2c1d1e5f1g1h3i1j1".
So obviously its best for all frames, back then it would have been a exception for large ones.
however WW's only compresses 0x0 entries meaning

0 - The transparent index
* - some other index

0 0 0 0 0
0 * * * 0
0 * * * 0
0 0 0 0 0

With WW's RLE compresses as

0 [5] * * * 0 [1]
* * * 0 [1]
* * * 0 [5]

[] - zeroes to insert when decompressing

There is no other compression than RLE, the Flags entry is a bitfield,
dunno what bit 1 is, bit 2 is IsCompressed, and rest i don't know.

you check if a bit is true like Flags & 2 which tells you IsCompressed.

Careful tho, apparently some old ShapeSet version WW had stored the Flags as a word and wrote it as a dword, result of that is like in harvestr.shp, where its 0xCCCC0003. So of all flags only the second word entries are valid.

Afict the other flags aren't checked in the game but the code is big mess so i can't be sure

As for PCX, its a vary simple format, you can find standalone implementations for C# too im sure, most complex thing in it would be compression with is well well well, RLE but its more or less RLE i described at first not just zeroes.
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Wed Dec 13, 2017 10:52 pm    Post subject: Reply with quote

yeah, at the start of this project i checked the PCX format. I only found a rough description on wikipedia and then the only free .NET library via ImageMagick.

Though i guess now with Nyerguds RLE routine and some time i could get PCX format done. However, apart from XCC Mixer which supports this as a secondary extraction format (PNG is also supported), i don't see why PCX would be necessary.
Especially since free tools like Irfanview can convert PNG to PCX and vice versa in a batch process, if someone really has old PCX lying around.


While researching a bit about SHP, i've read this topic where you among others discussed the format. Very interesting read btw. Approved!
Are there any new things found out?
Do i have to add an option for these strange Blitter bit flags? Any more info how or if they work?


Currently i write 01 when uncompressed and 03 when RLE_Zero compressed. From what i've read it should be actually 00 and 02.
However nearly all TS SHPs use 01 and 03.
I'm confused what is correct now. Confused
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Wed Dec 13, 2017 11:12 pm    Post subject: Reply with quote

As I understand it, we have no idea what the first bit actually does, though as you say its is often set.

Although it takes up 4 bytes in the file, we'll assume only the first one matters (because for our purposes it is the only thing that does). For example in binary these are the 4 values you can see:

00000000 = 0
00000001 = 1
00000010 = 2
00000011 = 3

If we treat each bit as a separate boolean value, then 3 is what you get when both bools are set to true. We know the second bool indicates compression, so both 2 and 3 are valid indicators of that since they both have the second bit set. We don't know what the first one does (if anything) in the final game, but its set when the value is either 1 or 3. So basically 0 or 1 for uncompressed are valid and 2 and 3 for compressed are valid.

Back to top
View user's profile Send private message
tomsons26lv
Scorpion Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Wed Dec 13, 2017 11:37 pm    Post subject: Reply with quote

Not much besides what i mention there.
Fun trivia on WW's RLE, is we found the RLE encoder in TS, its used to pack voxel shadows.

The game generates RLE compressed shadows for most if not all facings on the first time a voxel unit appears and makes a turn then the game caches them, then it only generates compressed shadows on slopes, not sure if those are cached as well, didn't seem to be, rle was called every time afict.

So yea bitfields, how you check them the crude way is bitfield & value, like
bitfield & 2 tells you if its compressed,
bitfield & 1 tells you if flag 1 is set whatever it is,
bitfield & 32 tells you if flag 6 is set

as for a byte bitfield its

bit | number value

1  |          1
2  |          2
3  |          4
4  |          8
5  |        16
6  |        32
7  |        64
8  |      128
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Thu Dec 14, 2017 2:49 pm    Post subject: Reply with quote

Version 01.01.00.06 uploaded
-a few adjustments on the class for the internal stored images/frames
-progressbar is showing the current progress of the imported files


Feel free to test again E1. I just successfully drag&dropped 65535 files (my public carrier frames copied several times) in one go into the tool.
Tool went up to 250MB memory usage max for all these in the datagrid.

Then pressed convert, memory usage went up to 4.2 Gigabyte (tested on a 64bit machine)
SHP created, size 1.27 GB (1.364.399.848 Bytes)
Duration: 850,0sec

Please no one ask me to upload and make this SHP public Wink

P.S.: SHP Builder can't open it
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
E1 Elite
Energy Commando


Joined: 28 May 2013

PostPosted: Thu Dec 14, 2017 3:13 pm    Post subject: Reply with quote

Nice, now it loads all 64k files in one go even when palette is loaded.
Never took more than 2.4 gb for the whole test which I was testing with.

Could mention that SHP files also could be dragged and dropped. Either
Load Images could be named as Load Images/SHP or could mention in
the readme info.

Back to top
View user's profile Send private message
tomsons26lv
Scorpion Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Thu Dec 14, 2017 4:16 pm    Post subject: Reply with quote

Right gotta post a rage about this

So you got
namespace ImageShaper
{
public enum SHP_TS_EncodingFormat
{
Undefined,
Uncompressed,
RLE_Zero,
Detect_best_size,
Unknown0,
Unknown2
}
}

switch (array[i].Format)
{
case 0:
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.Unknown0;
break;
case 1:
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.Uncompressed;
break;
case 2:
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.Unknown2;
break;
case 3:
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.RLE_Zero;
break;
default:
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.Undefined;
break;
}
----------------------------------------------------
No just no...
So

//assuming its a byte bitfield we get
enum Shape_TS_Frame_Flags
{
UnknownBit1  = 1,
IsCompressed = 2,
UnknownBit3  = 4,
UnknownBit4  = 8,
UnknownBit5  = 16,
UnknownBit6  = 32,
UnknownBit7  = 64,
UnknownBit8  = 128,
}
these are defined like this so you don't have a cryptic bitfield & 128


your weird switch translates to

if(array[i].Format & UnknownBit3)
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.Unknown0;
}

if(array[i].Format & UnknownBit1)
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.Uncompressed;
}
   
//this is wrong
if(array[i].Format & (UnknownBit1 | UnknownBit3) // this is what a number 5 is, bit 1 and 3;
{
   cimageFile.CompressionFormat = SHP_TS_EncodingFormat.Unknown2;  
}
   
if(array[i].Format & IsCompressed)
cimageFile.CompressionFormat = SHP_TS_EncodingFormat.RLE_Zero;
}

//0 doesn't exist in a bitfield, it just means no bits are set

---------------------------------------------------

Proper way to go about this is


//to make it simpler we just use bools, could reuse bitfield but C# sucks at bitfields
//these would be used in your frame reading code

bool FrameBool1;
bool FrameCompressed;
bool FrameBool3;
bool FrameBool4;
bool FrameBool5;
bool FrameBool6;
bool FrameBool7;
bool FrameBool8;

if(array[i].Format & UnknownBit3)
{
FrameBool3 = true;
}

if(array[i].Format & UnknownBit1)
{
FrameBool1 = true;
}

if(array[i].Format & IsCompressed)
{
FrameCompressed = true;
}

now in your frame processing code you use those bools in SHP_Frame_Bools,
i'd turn CompressionFormat in to a bitfield and check the bits as processing but whatever
-----------------------------------------------------

now to set a bitfield for when writing the shape frame

//setting bit 1
FrameFlags =| UnknownBit1;

//Setting bit 1 and 2
FrameFlags =| (UnknownBit1 | FrameCompressed);
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Thu Dec 14, 2017 4:50 pm    Post subject: Reply with quote

thx for the constructive rage. Smile

yes, i have to overhaul this entire bit flag part as this is pretty incompatible with the current way of handling the possible ways to write a frame (which were only 2 when i started this project).
Though not sure if worth the work, if it's only these 2 ways in the first place and all these unknown bits might be just useless.
But for sake of completeness i'll add the bit flag mechanism anyway and also add a way to allow the user to set/read them. Then maybe only in a "pro mode" so the normal user in "simple mode" doesn't have to care about these strange flags and only has the 3 options, uncompressed, RLE_Zero, detect_best_size.
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
tomsons26lv
Scorpion Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Thu Dec 14, 2017 5:21 pm    Post subject: Reply with quote

They may be useless they may be not, but they are there, checking as Flags & 2 ensures you know if a shp is rle compressed for all 3 known to be used numbers which are 2, 3 and 5.

One other note in Frame header after what you call RadarColor there's a dword what XCC called Zeros, its not always zero as some would claim, RA2's money.shp and tstlexp.shp have 00FB1200, no clue whats going on there.
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Last edited by tomsons26lv on Fri Dec 15, 2017 1:26 am; edited 1 time in total

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Thu Dec 14, 2017 8:35 pm    Post subject: Reply with quote

is it 3 byte radarcolor or 4 (with alpha?) and then followed by the dword?

also i've read something about the very first byte in a SHP not always being 00. Any more insight in that? Do i have to take care of it?
_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Thu Dec 14, 2017 9:04 pm    Post subject: Reply with quote

What your calling radar colour is not, its the average colour of the SHP file if it was to be viewed at a small size, what happens to be also used for the radar.

TS/RA2 does not use RGBA, only RGB, alpha is handled directly by a specifically blitter that takes the shape and blits it with a fixed transparency (25, 50, 75).

Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic Page 1 of 2 [56 Posts] Goto page: 1, 2 Next
View previous topic :: View next topic
 Forum index » Featured Tools » CnC Editing Tools
Jump to:  
Share on TwitterShare on FacebookShare on Google+Share on DiggShare on RedditShare on PInterestShare on Del.icio.usShare on Stumble Upon
Quick Reply
Username:


If you are visually impaired or cannot otherwise play the game below please contact the Administrator for help.


 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © phpBB Group

Wildcard SSL Certificates
[ Time: 0.4126s ][ Queries: 14 (0.0160s) ][ Debug on ]