:: Home :: Get Hosted :: PPM FAQ :: Forum FAQ :: Search :: Memberlist :: Usergroups :: Register :: Profile :: Log in to check your private messages :: Log in ::
Do you want to advertise at Project Perfect Mod. Find out how to do it HERE.

The time now is Thu Jun 29, 2017 4:04 pm
All times are UTC + 0
 Forum index » Featured Tools » OS SHP Builder
SHP
Moderators: stucuk
Post new topic   Reply to topic Page 2 of 3 [148 Posts] View previous topic :: View next topic
Goto page: Previous 1, 2, 3 Next
Author Message
ApolloTD
Commander


Joined: 19 Nov 2003

PostPosted: Mon Aug 22, 2016 5:36 am    Post subject: Reply with quote

04 00 00 00 is for blank frames clearly judging by WW shps and the ww tool test and not surprised it glitches when give content.

fog.shp/shroud.shp can't be compressed anyway, its an alpha process and those do not have compressed blitter variant so misdraws if was compressed.
Not sure why you thought its compressed TBH.

BTW, the stripe bug is AFAIK unfixable in the shp itself as game itself has a hard coded limit stupidly set in place (ARES fixes this WW programmer oversight) for the compressed line and past that reads stuff normally than process compressed as usual.

00 is same as 01, no difference ingame.

02, there is not single shp in the game I have ever seen using it and would gladly wanna see any official shp that has it thus likely invalid.

Frankly as interesting this 'poking randomly' is, there is not likely any improved compressions I'd be interested in achieved here as RLE compression method by WW is inefficient sadly as even simple PCX's RLE does better if give it same cropped source!

It is pity they did not bring LCW to TS for good compress TBH or at very least RLE the content as well and forget stupid line lenghts declaring which only wastes space.

Back to top
View user's profile Send private message
MadHQ
Commander


Joined: 07 Nov 2003

PostPosted: Mon Aug 22, 2016 7:33 am    Post subject: Reply with quote

Ah! the fun of de-compelling file formats... XP

I remember the fun Hell I had just getting this to work.

http://zombapro.ppmsite.com/experiment/sphdrawNEW.php

All of ngcnstdm.shp file data. XP

http://zombapro.ppmsite.com/experiment/readBuildshow.php

I should really work on that stuff more...

and

I should really join in on this conversation an help... Maybe...
_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Back to top
View user's profile Send private message MSN Messenger
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Mon Aug 22, 2016 7:41 am    Post subject: Reply with quote

ApolloTD wrote:
02, there is not single shp in the game I have ever seen using it and would gladly wanna see any official shp that has it thus likely invalid.


RA2\ra2\IsoSnow\cafncb.shp
RA2\ra2\IsoTemp\ctfncb.shp
RA2\ra2\IsoUrb\cufncb.shp

Fence Post's use it. Don't see why since 2 is the same as 1 except you have 2 bytes extra per row to tell it the width you already know.

Spoiler (click here to read it):
RA2\ra2\IsoSnow\cafncb.shp 68x66 Count: 64
Frame 0000:   33,  37    2x  11   2 02000000 015,015,015,000
Frame 0001:   34,  30   15x  18   3 03000000 015,015,015,000
Frame 0002:   34,  38   15x  17   3 03000000 016,016,016,000
Frame 0003:   34,  30   15x  25   3 03000000 015,015,015,000
Frame 0004:   19,  38   15x  17   3 03000000 016,016,016,000
Frame 0005:   19,  30   30x  25   3 03000000 015,015,015,000
Frame 0006:   19,  38   30x  17   3 03000000 016,016,016,000
Frame 0007:   19,  30   30x  25   3 03000000 015,015,015,000
Frame 0008:   19,  30   15x  18   3 03000000 015,015,015,000
Frame 0009:   19,  30   30x  18   3 03000000 015,015,015,000
Frame 0010:   19,  30   30x  25   3 03000000 015,015,015,000
Frame 0011:   19,  30   30x  25   3 03000000 015,015,015,000
Frame 0012:   19,  30   16x  25   3 03000000 015,015,015,000
Frame 0013:   19,  30   30x  25   3 03000000 015,015,015,000
Frame 0014:   19,  30   30x  25   3 03000000 015,015,015,000
Frame 0015:   19,  30   30x  25   3 03000000 015,015,015,000
Frame 0016:   33,  38    2x   9   3 03000000 003,003,003,000
Frame 0017:   34,  31   15x  16   3 03000000 009,009,009,000
Frame 0018:   34,  38   15x  16   3 03000000 009,009,009,000
Frame 0019:   34,  31   15x  23   3 03000000 009,009,009,000
Frame 0020:   19,  39   15x  16   3 03000000 009,009,009,000
Frame 0021:   19,  31   30x  24   3 03000000 009,009,009,000
Frame 0022:   19,  38   30x  17   3 03000000 009,009,009,000
Frame 0023:   19,  31   30x  24   3 03000000 008,008,008,000
Frame 0024:   19,  30   15x  17   3 03000000 009,009,009,000
Frame 0025:   19,  30   30x  17   3 03000000 010,010,010,000
Frame 0026:   19,  30   30x  24   3 03000000 009,009,009,000
Frame 0027:   19,  30   30x  24   3 03000000 009,009,009,000
Frame 0028:   19,  30   16x  25   3 03000000 009,009,009,000
Frame 0029:   19,  30   30x  25   3 03000000 010,010,010,000
Frame 0030:   19,  30   30x  25   3 03000000 009,009,009,000
Frame 0031:   19,  30   30x  25   3 03000000 009,009,009,000
Frame 0032:   35,  46    3x   2   2 02000000 000,000,000,000
Frame 0033:   35,  40   16x   9   3 03000000 000,000,000,000
Frame 0034:   34,  45   18x  10   3 03000000 000,000,000,000
Frame 0035:   34,  40   18x  15   3 03000000 000,000,000,000
Frame 0036:   19,  48   16x   8   3 03000000 000,000,000,000
Frame 0037:   19,  40   32x  16   3 03000000 000,000,000,000
Frame 0038:   19,  46   33x  10   3 03000000 000,000,000,000
Frame 0039:   19,  40   33x  16   3 03000000 000,000,000,000
Frame 0040:   20,  39   15x   9   3 03000000 000,000,000,000
Frame 0041:   20,  39   31x   9   3 03000000 000,000,000,000
Frame 0042:   20,  39   32x  16   3 03000000 000,000,000,000
Frame 0043:   20,  39   32x  16   3 03000000 000,000,000,000
Frame 0044:   19,  39   18x  17   3 03000000 000,000,000,000
Frame 0045:   19,  39   32x  17   3 03000000 000,000,000,000
Frame 0046:   19,  39   33x  17   3 03000000 000,000,000,000
Frame 0047:   19,  39   33x  17   3 03000000 000,000,000,000
Frame 0048:   34,  45    1x   1   2 02000000 000,000,000,000
Frame 0049:   35,  40   16x   9   3 03000000 000,000,000,000
Frame 0050:   34,  45   18x  10   3 03000000 000,000,000,000
Frame 0051:   34,  40   18x  15   3 03000000 000,000,000,000
Frame 0052:   19,  48   16x   8   3 03000000 000,000,000,000
Frame 0053:   19,  40   32x  16   3 03000000 000,000,000,000
Frame 0054:   19,  46   33x  10   3 03000000 000,000,000,000
Frame 0055:   19,  40   33x  16   3 03000000 000,000,000,000
Frame 0056:   20,  39   13x   6   3 03000000 000,000,000,000
Frame 0057:   20,  39   31x   9   3 03000000 000,000,000,000
Frame 0058:   20,  39   32x  16   3 03000000 000,000,000,000
Frame 0059:   20,  39   32x  16   3 03000000 000,000,000,000
Frame 0060:   19,  39   16x  17   3 03000000 000,000,000,000
Frame 0061:   19,  39   32x  17   3 03000000 000,000,000,000
Frame 0062:   19,  39   33x  17   3 03000000 000,000,000,000
Frame 0063:   19,  39   33x  17   3 03000000 000,000,000,000


---------------------------------------------------------------------------------

Added SHP Builder's SHP Type/Game detection (Which has alot of holes in it... except for cameos which it mostly gets right).


_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Mon Aug 22, 2016 8:49 am    Post subject: Reply with quote

Implemented a "Ignore Boring Frames" edition of the debug. Now with less boring data.

Basically frames (And files) which are the bog standard are ignored. So any frame which either has no Width/Height or which has compression as 1,2,3 and zeros for the other 3 bytes. Only the interesting ones are left.

See Attachment.


SHPs_debug.txt
 Description:

Download
 Filename:  SHPs_debug.txt
 Filesize:  334.74 KB
 Downloaded:  3 Time(s)


_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Mon Aug 22, 2016 1:03 pm    Post subject: Reply with quote

Implemented Shadow Support. Supports shadows on SHP's that are classed as Unit, Building or Building Animation. Shadows can either be rendered in their normal colour or with partial transparency (And black colour). Shadows will always be shown when you hit the animate button or when your using the Open File Dialog (However the bit that lists shps can only do RA2 units since it only loads a single frame).

Implemented the remaining TD/RA1 decode code (Aka nicked the Decode40Tri from SHP Builder). Currently has no ability to save TD/RA1 SHP's.

Also updated the Side Colour code to apply the colour to the correct RA1 palette location.



P.S I added code to check if a Unit is a RA2 unit by scanning the first frame for shadow colours (Since TS never has shadows in the first frame, nor the same shadow colour). So it should always work out TS/RA2 units correctly (At least official ones). Also added code to fix buildings getting the Animation Type by checking for shadows.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Mon Aug 22, 2016 2:39 pm    Post subject: Reply with quote

Have you ensured that other tools correctly read your files as well? You idea to add a tailer that contains palette and other information is probably a bad idea regards compatibility with other tools, I know that XCC validates the file size for TD/RA shape files at least (though you aren't writing those yet).

You should also probably look at the improvements I made to the TD/RA Shape encoder in XCC for implementing that format.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Mon Aug 22, 2016 2:41 pm    Post subject: Reply with quote

stucuk wrote:
P.S I added code to check if a Unit is a RA2 unit by scanning the first frame for shadow colours (Since TS never has shadows in the first frame, nor the same shadow colour). So it should always work out TS/RA2 units correctly (At least official ones). Also added code to fix buildings getting the Animation Type by checking for shadows.

IIRC only RA2 infantry used shadow on the first frame, but then not index #1 but index #12 (pure black) color.
TS and RA2 use both palette index #1 for shadow (#0 being transparent). Since the real color plays no role and depends on the loaded palette, i don't see how you want to determine the shadow color only by looking at the SHP, nor how this allows you to find out for which game the SHP was made. Confused

You can use an SHP made for TS and show it exactly the same in RA2, if RA2 has the same palette. So from an SHP point there is absolutely no difference between the games.
_________________
SHP Artist of Twisted Insurrection. Main Nod building SHP creator.

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant unit, 1 Scrin unit, 1 GDI building

Tools
TMP Shop
C&C Executable Modifier

Back to top
View user's profile Send private message
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Mon Aug 22, 2016 7:56 pm    Post subject: Reply with quote

- SHPToolz_2016-08-22.rar

Note: I haven't built the debug build. So ignore it. Also may have some memory leaks as i haven't built the debug build to check.

Blade wrote:
Have you ensured that other tools correctly read your files as well? You idea to add a tailer that contains palette and other information is probably a bad idea regards compatibility with other tools, I know that XCC validates the file size for TD/RA shape files at least (though you aren't writing those yet).


As long as the games work fine with it i'm happy. I won't be dictated to by 3rd party tools which were designed badly(If people "Obey" the file format there should be no issues). However i do plan on making the extra data being saved optional when i do a custom save dialog (There is already a FSaveExtra in the TSHP class which determines if it saves the extra data).

Lin Kuei Ominae wrote:
IIRC only RA2 infantry used shadow on the first frame, but then not index #1 but index #12 (pure black) color.
TS and RA2 use both palette index #1 for shadow (#0 being transparent). Since the real color plays no role and depends on the loaded palette, i don't see how you want to determine the shadow color only by looking at the SHP, nor how this allows you to find out for which game the SHP was made. Confused


Code:
if (FType = stUnit) and (FGame = sgTS) then
 if ScanFrameForColor(12,0) then
 FGame := sgRA2;

 if (FType = stAnimation) then
 if ScanFrameForColor(1,(High(FFrames)+1) div 2) then
 FType := stBuildAnim;


Just checks for Colour #12 on the first frame if its classed as a unit. Currently only treats #12 as the shadow colour if its a RA2 SHP and has the Unit Type. Everything else with TS/RA2 it uses colour #1. With RA1 it uses #4. I havn't extracted TD files yet so it currently ignores TD.

Lin Kuei Ominae wrote:
You can use an SHP made for TS and show it exactly the same in RA2, if RA2 has the same palette. So from an SHP point there is absolutely no difference between the games.


Which is why it tries to determine the Game as well as Type. You then have a higher chance of automatically picking the correct palette.

When animating the Type comes into play as Units/Buildings/BuildAnims all have shadow frames to skip when animating (As in consider the end of the animation where the shadow frames start).
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Mon Aug 22, 2016 8:11 pm    Post subject: Reply with quote

from where do you get FType and FGame? There are no such variables in the SHP afaik, except you mean already your added extra data from the end of the file.

That whole "guessing" already made SHP Builder quite a pain to work with, until Banshee added the option to remember the last used palette and thus allowed to disable the not working SHP type guessing.
It's simply that the guessing doesn't and can't work well enough.
Especially not with all the custom made SHPs.
And when you are later forced to constantly select your working palette, because the program guessed a wrong one again, this can become quite annoying.
When you are working with a tool, it's a lot more likely that you keep using the same chosen palette, than constantly a different one.

Not to mention that nowadays we use 90% custom palettes. So the whole guessing can't work at all anymore.
_________________
SHP Artist of Twisted Insurrection. Main Nod building SHP creator.

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant unit, 1 Scrin unit, 1 GDI building

Tools
TMP Shop
C&C Executable Modifier

Back to top
View user's profile Send private message
tomsons26lv
Pyro Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Mon Aug 22, 2016 9:11 pm    Post subject: Reply with quote

ApolloTD wrote:

BTW, the stripe bug is AFAIK unfixable in the shp itself as game itself has a hard coded limit stupidly set in place (ARES fixes this WW programmer oversight) for the compressed line and past that reads stuff normally than process compressed as usual.

No clue what Ares fixed, but changing the Flags actually affects that, which is why i started to think its controlling Blitters in some way in the first place. I managed to break it so much the units behind the building were rendering all on top.

I'm starting to think Buildingz comes to play somewhere...
Well i do have reasonable cause to think it. ShapeSet uses the same Blitting functions as the game, this is in one of them



ApolloTD wrote:
04 00 00 00 is for blank frames clearly judging by WW shps and the ww tool test and not surprised it glitches when give content.

fog.shp/shroud.shp can't be compressed anyway, its an alpha process and those do not have compressed blitter variant so misdraws if was compressed.
Not sure why you thought its compressed TBH.

BTW, the stripe bug is AFAIK unfixable in the shp itself as game itself has a hard coded limit stupidly set in place (ARES fixes this WW programmer oversight) for the compressed line and past that reads stuff normally than process compressed as usual.

00 is same as 01, no difference ingame.

02, there is not single shp in the game I have ever seen using it and would gladly wanna see any official shp that has it thus likely invalid.

Frankly as interesting this 'poking randomly' is, there is not likely any improved compressions I'd be interested in achieved here as RLE compression method by WW is inefficient sadly as even simple PCX's RLE does better if give it same cropped source!

It is pity they did not bring LCW to TS for good compress TBH or at very least RLE the content as well and forget stupid line lenghts declaring which only wastes space.


The point of that image was to show that non-compressed shps can be viewed as raw data, and the point of that is was to offer a different perspective on the files removing the factor of the over a decade old baseless assumptions that compression is specified in the shp header.

There isn't, there's just RLE, it's possible there might be variations of RLE not just RLE Zero, but from what i saw when Bittah tested the flags i doubt that.
WW made in the way they needed it. The RLE just compresses Zeros, as far as its efficiency goes it does it properly.
ShapeSet doesn't show any proof of any other compressions either.
Hyper looked into that hidden -X command that set flags to 1 and added extra two bytes with 00's in my test shp, it would seem it's related to shp's created from DPaint Animation files.
Yea as far as LCW goes its in the exes but its only used for VQA's and some legacy code that the game itself no longer uses.
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Last edited by tomsons26lv on Mon Aug 22, 2016 11:28 pm; edited 2 times in total

Back to top
View user's profile Send private message Visit poster's website
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Mon Aug 22, 2016 9:29 pm    Post subject: Reply with quote

stucuk wrote:


Blade wrote:
Have you ensured that other tools correctly read your files as well? You idea to add a tailer that contains palette and other information is probably a bad idea regards compatibility with other tools, I know that XCC validates the file size for TD/RA shape files at least (though you aren't writing those yet).


As long as the games work fine with it i'm happy. I won't be dictated to by 3rd party tools which were designed badly(If people "Obey" the file format there should be no issues). However i do plan on making the extra data being saved optional when i do a custom save dialog (There is already a FSaveExtra in the TSHP class which determines if it saves the extra data).



Much as some of the code in XCC isn't easy to follow, you can hardly claim its badly designed for not "Obeying" the file format when you yourself are blatantly planning on creating out of spec files. The general rule when creating tools to work with a file format is be a flexible as possible when reading them while being as strict as possible when writing them.

There is also the issue that this wastes memory in game as shape files are loaded into memory in their entirety. Not that its a massive issue with he GB's we have these days, but I would definitely say people shouldn't distribute shape files with the extra information in them for these reasons.

At the end of the day its your project and you can take it in whatever directions you want, but you already complained hardly anyone contributed to Shp Builder but being dismissive of other people's opinions and feedback and proclaiming that you won't be beholden to 3rd party utilities handling of the files doesn't sound to me that you are too concerned about getting other people on board with supporting it.

Back to top
View user's profile Send private message
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Mon Aug 22, 2016 10:47 pm    Post subject: Reply with quote

Lin Kuei Ominae wrote:
from where do you get FType and FGame? There are no such variables in the SHP afaik, except you mean already your added extra data from the end of the file.


Its not anywhere in the format... its part of the TSHP class.... Source code is included.

Lin Kuei Ominae wrote:
That whole "guessing" already made SHP Builder quite a pain to work with, until Banshee added the option to remember the last used palette and thus allowed to disable the not working SHP type guessing.
It's simply that the guessing doesn't and can't work well enough.
Especially not with all the custom made SHPs.
And when you are later forced to constantly select your working palette, because the program guessed a wrong one again, this can become quite annoying.
When you are working with a tool, it's a lot more likely that you keep using the same chosen palette, than constantly a different one.


Thats the whole point of the Extra data added to the SHP file. It saves all of it...

Lin Kuei Ominae wrote:
Not to mention that nowadays we use 90% custom palettes. So the whole guessing can't work at all anymore.


Saved as extra data....

Blade wrote:
Much as some of the code in XCC isn't easy to follow, you can hardly claim its badly designed for not "Obeying" the file format when you yourself are blatantly planning on creating out of spec files. The general rule when creating tools to work with a file format is be a flexible as possible when reading them while being as strict as possible when writing them.


Given how SHP files are layed out, there is no reason to read beyond the frame data. Any application which randomly tries to load unknown data (Or randomly assumes it has to be a certain bit of data) which isn't in the specification is badly coded. You should always follow the rules of a file format and never guess. You can take most file formats and add data at the end of them without issue (EXE's for example). XCC 1.47 and SHP Builder (Any version) doesn't seem to have any issues loading the SHP's (There is no reason for any tool to have an issue). Where is the phantom SHP tools which crash when they load a SHP with extra data?

Unless you find a tool which has an issue with extra data at the end of a shp then its pointless arguing about it.

Blade wrote:
There is also the issue that this wastes memory in game as shape files are loaded into memory in their entirety. Not that its a massive issue with he GB's we have these days, but I would definitely say people shouldn't distribute shape files with the extra information in them for these reasons.


778 Bytes extra data, equivalent to a frame with 27x27 pixels (Including the frame header). Your not seriously going to say that 778 Bytes extra per SHP is going to make a computer grind to a halt....

If you have 3,606 SHP files (Thats the total i have in my SHP folder containing SHP's from RA1, TS, RA2) you would have to load an extra 2.67MB into memory. 2.67MB is nothing even back in 2000. Your argument over increased size is completely irrelevant.

Blade wrote:
but you already complained hardly anyone contributed to Shp Builder but being dismissive of other people's opinions and feedback

What feedback have i been dismissive of? I take all feedback into account but i am not necessarily going to do everything that everyone wants.

If your referring to the "It may break stuff" then its not going to hold me back unless you find an actual case where its an issue. Everything is speculation and as it stands SHPToolz will not be used by ANYONE for anything as currently it can't do much other than view SHP's. The end of the world hasn't happened yet.

Blade wrote:
and proclaiming that you won't be beholden to 3rd party utilities handling of the files doesn't sound to me that you are too concerned about getting other people on board with supporting it.

People supporting the tool has nothing to do with another tools ability to load SHP's. As it stands there is no reason for any application that loads SHP files to do anything with any extra data found in a SHP file. Your making a big deal out of nothing.

Lets say that Tool "A" writes a SHP using format 999 which the game supports and has superior compression, but Tool "B" doesn't like format 999. Should Tool "A" drop support for format 999 simply because Tool "B" wasn't designed to handle it? Extra Data in this case should be ignored as there is no reason for a tool to load anything extra than what it expects. Something like Extra Data should be something other tools want to support so that SHP's can be rendered more properly across the board.

The main point of me writing the code is to build the TSHP class. It gives everyone the ability to load and save SHP files however they like and is full of options like not saving the Extra data to a SHP file. You can easily use it to create a tool which doesn't save extra data.


P.S SHPToolz can't actually read the Extra SHP data in the build i released as the code is wrong, it just skips it.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
Banshee
Supreme Banshee


Joined: 15 Aug 2002
Location: Brazil

PostPosted: Mon Aug 22, 2016 11:15 pm    Post subject: Reply with quote

Lin Kuei Ominae wrote:
Not to mention that nowadays we use 90% custom palettes. So the whole guessing can't work at all anymore.


Just to point out: in OS SHP Builder, you can customize the palettes that are loaded for units, infantry, animations, cameo, etc... for each game.

Back to top
View user's profile Send private message Visit poster's website MSN Messenger
ApolloTD
Commander


Joined: 19 Nov 2003

PostPosted: Tue Aug 23, 2016 5:44 am    Post subject: Reply with quote

stucuk wrote:
ApolloTD wrote:
02, there is not single shp in the game I have ever seen using it and would gladly wanna see any official shp that has it thus likely invalid.


RA2\ra2\IsoSnow\cafncb.shp
RA2\ra2\IsoTemp\ctfncb.shp
RA2\ra2\IsoUrb\cufncb.shp

Fence Post's use it. Don't see why since 2 is the same as 1 except you have 2 bytes extra per row to tell it the width you already know.
[/url]


Okay, I checked the relevant files, it basicly means, compression 2 is ONLY used when there are none of background pixels (0) within the cropped canvas but the benefit of this compression is pointless when you just tell linelenghts all for nothing wasting space :/

Anyways the extra trailer data will get read by OpenRA sadly as pixels, already had this problem with alphatst.shp way back when outside of canvas were extra junk bytes. Obviously TS/RA2 will have no issue as reads strictly to the demands of canvas size set as far drawing goes but it will probably store whole file in RAM.

tomsons26lv wrote:

The point of that image was to show that non-compressed shps can be viewed as raw data, and the point of that is was to offer a different perspective on the files removing the factor of the over a decade old baseless assumptions that compression is specified in the shp header.

There isn't, there's just RLE, it's possible there might be variations of RLE not just RLE Zero, but from what i saw when Bittah tested the flags i doubt that.
WW made in the way they needed it. The RLE just compresses Zeros, as far as its efficiency goes it does it properly.
ShapeSet doesn't show any proof of any other compressions either.
Hyper looked into that hidden -X command that set flags to 1 and added extra two bytes with 00's in my test shp, it would seem it's related to shp's created from DPaint Animation files.
Yea as far as LCW goes its in the exes but its only used for VQA's and some legacy code that the game itself no longer uses.


Yeah sure can view uncompressed that way but frankly I can tell even with just hex editor when they are compressed and not easily. There is no variations on the RLE and no, this RLE is not efficient in terms of filespace usage. You waste 2 bytes every time you tell there is a background pixel, then you waste 2 bytes to tell the line lenght per every line. Where as PCX RLE is encoded on the principle that every byte is assumed to be single byte telling RLE count, exceptional range value to this can be used to signal when to process normally and how many normal bytes follow before switch back to normal RLE parsing.

Thus to skip 50 background pixels, PCX basicly puts value 50 while shp puts 0 and 50. INEFFICIENT. Then lets assume width of frame was 50 pixels, shp also tells it in 2 bytes making it up to +3 waste bytes compared to pcx method Smile Outcome of this I seen is, shp (C3) ends up 2.10mb and pcx of same content 1.60mb...
That is what I mean inefficient, it sure works for the game but it is wasteful byte programming if really wanted to compress GOOD.

Yep LCW is in the exe but only for VQA actively, could have kept for SHPs but discarded since RA1 shps, likely performance issues with larger filesizes which was no issue for video files when only one gets called at a time instead of dozen.

Last edited by ApolloTD on Tue Aug 23, 2016 7:46 am; edited 1 time in total

Back to top
View user's profile Send private message
E1 Elite
Railgun Soldier


Joined: 28 May 2013

PostPosted: Tue Aug 23, 2016 7:29 am    Post subject: Reply with quote

SHP like RA2 ctbunk02.shp uses 2 palettes, unittem for image and isotem for rubble frame.
Your current extra data wouldn't account for that. Would be nice if such SHP are viewable
correctly. RA2 ctcity04.shp uses citytem palette which cannot be viewed by your tool right
now as it is not included.

Custom palette support in dropdowns is like a need for mods. Suggestion is to scan the folder
for palettes and the usage of palette names as they are like unittem instead of using fixed
names like Building or Unit.

Back to top
View user's profile Send private message
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Tue Aug 23, 2016 7:46 am    Post subject: Reply with quote

ApolloTD wrote:
Anyways the extra data will get read by OpenRA sadly as pixels, already had this problem with alphatst.shp way back when outside of canvas were extra junk bytes.

Looking at the Source code i don't see that. Maybe i am looking at the wrong code but according to the following code they load SHP's "Properly"(As in sticking to the specification).

- ShpTDLoader.cs
- ShpTSLoader.cs

ApolloTD wrote:

That is what I mean inefficient, it sure works for the game but it is wasteful byte programming if really wanted to compress GOOD.

I would imagine they were going for a trade off between compression and rendering speed. Rendering a simple RLE like SHP's have will be faster than a proper one.

E1 Elite wrote:
SHP like RA2 ctbunk02.shp uses 2 palettes, unittem for image and isotem for rubble frame.
Your current extra data wouldn't account for that.

It would if a new type was added specifically for it. Selecting that type would make it use two palettes. Though i'm guessing that the rubble frame is at the same index for each building? Otherwise it would be impossible.

E1 Elite wrote:
RA2 ctcity04.shp uses citytem palette which cannot be viewed by your tool right
now as it is not included.

At this stage i haven't gone through the mix's for all the palettes. Though citytem.pal should be in the latest download (Its just never used by the app).

E1 Elite wrote:
usage of palette names as they are like unittem instead of using fixed
names like Building or Unit. It would also remove any doubt whether I am using the correct
palette.

Well the Type part is not entirely just for the palette(Its only tied to the palette in relation to automatically working out the default palette for that SHP before the user selects a custom one), its also for example to work out how to animate it as buildings/units have the separate shadow frames. Everything is a work in progress however.

There will be the ability to select a custom palette (Its just not coded yet) and it will display the palette name.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
E1 Elite
Railgun Soldier


Joined: 28 May 2013

PostPosted: Tue Aug 23, 2016 9:15 am    Post subject: Reply with quote

Usually rubble frame is 4th in the index, but if you can mark it as the last normal frame (not
shadow) it would be better.

Problem with the fixed palette name is that it lacks theater info which creates a doubt whether
we are using the correct palette. Moreover multiple of those crowding the dropdown like Unit,
Building, Building Anim etc. refer to the same palette and we are used to seeing unittem.pal
in there.

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Tue Aug 23, 2016 11:52 am    Post subject: Reply with quote

ModEnc seems to agree that there is a flag in the header that indicates either compression (Zero RLE) or no compression and that is bit 2 of the flags (basically and & 0x02 check).

stucuk, you complain that other tools shouldn't assume the end of the image data is the end of the file, but as it exists, that IS the spec, they aren't out of spec by doing that. Infact in the TD format header, the last image offset is to the end of the file, so its perfectly within spec to check that the flile end and that offset agree. Its you who are proposing to arbitrarily and unilaterally extend the spec and expect all other tools to play nicely with it.

Basically this boils down to the SHP format not supporting the meta data you want to associate with it and you are exploiting the fact that the game doesn't do any strict conformance checking in the same way the mix "protectors" corrupt the mix format.

A more compatible solution would be to generate a meta data file named the same as the shp file that can be distributed with it IMO (ed SPRITE.SHP gets SPRITE.TLZ or some other extension generated to go with it). That way the actual files are kept clean, your tool can have its own specialised meta data to work with and other tools will always ignore it as you intend them to.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Tue Aug 23, 2016 1:29 pm    Post subject: Reply with quote

Blade wrote:
generate a meta data file named the same as the shp file that can be distributed with it

I would support that.
_________________
SHP Artist of Twisted Insurrection. Main Nod building SHP creator.

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant unit, 1 Scrin unit, 1 GDI building

Tools
TMP Shop
C&C Executable Modifier

Back to top
View user's profile Send private message
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Tue Aug 23, 2016 3:11 pm    Post subject: Reply with quote

Blade wrote:
ModEnc seems to agree that there is a flag in the header that indicates either compression (Zero RLE) or no compression and that is bit 2 of the flags (basically and & 0x02 check).

What does 0x01 mean then? Cos its used with both no compression and Zero RLE but isn't set when its "Decode 2".

Blade wrote:
stucuk, you complain that other tools shouldn't assume the end of the image data is the end of the file, but as it exists, that IS the spec, they aren't out of spec by doing that.


The specification is the layout of the file based on what should exist given the header's data. There is nothing to state that the EOF is just after the last frame and nothing to state it isn't.

Answer me this one question. Which applications fail to load the extra data? Until you answer that question its only an issue of your making.

Blade wrote:
Basically this boils down to the SHP format not supporting the meta data you want to associate with it and you are exploiting the fact that the game doesn't do any strict conformance checking in the same way the mix "protectors" corrupt the mix format.


SHP Format doesn't specify anything after it. Thats the point. You only read what is specified and there is no reason to care if extra data is after it as you have already loaded everything the headers specified. Why can't you understand that simple fact.

If your writing good code you should never assume your reading a single file. For all your code knows it could be passed a stream that contains multiple files.

Corrupting a header is completely different to adding data AFTER the regular file. The former is designed to harm and the latter should be ignored by everything that doesn't know what it is (Since nothing in the Spec states the "BodySize" unlike the Mix Spec. Apples and Oranges).

P.S I haven't checked XCC's source, but it should just be making sure that BodySize is not larger than the amount of data in the file (Smaller should be no issue).

Blade wrote:
Infact in the TD format header, the last image offset is to the end of the file, so its perfectly within spec to check that the flile end and that offset agree.


The last offset is not to the end of the file but to the last image data (at least based on official TD SHP files). There is no reason to read beyond it or check that after reading the last frame that its the EOF. You already loaded the entire SHP file according to what the Headers told you.

If the game doesn't do a random check then why should you? Its got nothing to do with the specification to check where the EOF is as it doesn't specify where it is.

P.S ccfiles4.txt is an example of what Specifications are. They tell you what to load and how to load it. It states nothing on where the EOF is in relation to the last frame. As long as people follow the specification and don't add their own random stuff like seeing where the EOF is then there will never be any issues.

Lin Kuei Ominae wrote:
I would support that.

So would i if it was an issue. As it stands its not. I have still to hear about an application which can't handle it. So far its just Blade speculating that some random application may not be able to handle it.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Tue Aug 23, 2016 3:20 pm    Post subject: Reply with quote

stucuk wrote:
Lin Kuei Ominae wrote:
I would support that.

So would i if it was an issue. As it stands its not. I have still to hear about an application which can't handle it.

Doesn't matter for me. I would still prefer a clean SHP with an extra data additional file.
_________________
SHP Artist of Twisted Insurrection. Main Nod building SHP creator.

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant unit, 1 Scrin unit, 1 GDI building

Tools
TMP Shop
C&C Executable Modifier

Back to top
View user's profile Send private message
tomsons26lv
Pyro Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Tue Aug 23, 2016 3:55 pm    Post subject: Reply with quote

stucuk wrote:
Blade wrote:
ModEnc seems to agree that there is a flag in the header that indicates either compression (Zero RLE) or no compression and that is bit 2 of the flags (basically and & 0x02 check).

What does 0x01 mean then? Cos its used with both no compression and Zero RLE but isn't set when its "Decode 2".

I have more answers on that now actually,the first byte?(could as well be a word or even dword bitfield as WW did already use those in RA) a bitfield, see the txt i keep posting but seemingly you keep missing.

stucuk wrote:

Answer me this one question. Which applications fail to load the extra data? Until you answer that question its only an issue of your making.

I tried adding extra data after the shp, as it stands i only tested it for a moment, in RA and RA2 it works, XCC says the shps are unknown, OpenRA crashes on game startup.
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Tue Aug 23, 2016 4:00 pm    Post subject: Reply with quote

It is quite possible that there is no code in the game for Flag type 1, with Flag 0 being No compression and Flag 2 being Zero RLE.

The game only checks the header flag 'just' before blitting and it only checks for Flag 2, otherwise, No compression is assumed (Being 0 or 1).

I would have to investigate Shapeset more to find out how it claculates the Flags when writing a shape file.

EDIT: Due to tomsons testing, 0 is definatly No compression, 1 is ignored in TS and it seems RA2/YR has added code to handle Flag 1. No update as to what it is actually for yet.

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Tue Aug 23, 2016 9:37 pm    Post subject: Reply with quote

stucuk wrote:

Blade wrote:
Infact in the TD format header, the last image offset is to the end of the file, so its perfectly within spec to check that the flile end and that offset agree.


The last offset is not to the end of the file but to the last image data (at least based on official TD SHP files). There is no reason to read beyond it or check that after reading the last frame that its the EOF. You already loaded the entire SHP file according to what the Headers told you.

If the game doesn't do a random check then why should you? Its got nothing to do with the specification to check where the EOF is as it doesn't specify where it is.

P.S ccfiles4.txt is an example of what Specifications are. They tell you what to load and how to load it. It states nothing on where the EOF is in relation to the last frame. As long as people follow the specification and don't add their own random stuff like seeing where the EOF is then there will never be any issues.


That specification is incomplete, all the official shape files have frames + 2 entries in the offset array. If there is a "loop frame" then the second to last points to the start of it and the last entry points to the end of the file. If no loop frame is present (most shape files), then the second to last points to the end of the file and the last is 0's. I've done a lot of work on the TD shape format, I assure you this spec is correct.

XCC checks this to determine if a file is a TD Shape file BTW.

I don't get why you are so invested in attaching the data to the end of the file when to any other application is just going to be extraneous junk at best, why not just keep it separate?

Back to top
View user's profile Send private message
Banshee
Supreme Banshee


Joined: 15 Aug 2002
Location: Brazil

PostPosted: Tue Aug 23, 2016 9:47 pm    Post subject: Reply with quote

Stucuk, you should stop being stubborn for a while and listen to Blade's advise. Add aditional data to an external file as he says. Do not expect people in the future may not necessarily follow your standards, even if they happen to know and use your program.

Back to top
View user's profile Send private message Visit poster's website MSN Messenger
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Wed Aug 24, 2016 1:09 am    Post subject: Reply with quote

tomsons26lv wrote:
I tried adding extra data after the shp, as it stands i only tested it for a moment, in RA and RA2 it works, XCC says the shps are unknown, OpenRA crashes on game startup.

Are you adding it to TD/RA1 SHP's or TS/RA2 SHP's? XCC states the type as "shp (TS)" when extra data is saved to a TS/RA2 SHP. Based on what blade has said if its a TD SHP it will fail if you only add extra data (Though altering the eof offset may make it work. I still need to get round to coding the TD SHP writer).

CCHyper wrote:
It is quite possible that there is no code in the game for Flag type 1, with Flag 0 being No compression and Flag 2 being Zero RLE

Ah, that makes so much sense. "Decode 2" Is "Decode 3" but it just has no zeros in it. Though i don't get why their encoder would choose to compress it when uncompressed would be smaller. There are SHP's which have both compressed and uncompressed frames in the same SHP.

@Banshee: SHP Builder doesn't treat it like a flag.

Blade wrote:
That specification is incomplete, all the official shape files have frames + 2 entries in the offset array. If there is a "loop frame" then the second to last points to the start of it and the last entry points to the end of the file. If no loop frame is present (most shape files), then the second to last points to the end of the file and the last is 0's. I've done a lot of work on the TD shape format, I assure you this spec is correct.

Then what would happen if you modify the eof offset to be the true eof. As in after the extra data rather than last frame. Also it may be possible to store data after the offsets and before the first frame starts (Depending on if it looks at the first frames offset and seeks to it or if it just starts where it left off. If the former then it shouldn't be an issue).

The question i have is if the first extra offset is where it should go after reaching the end of the animation then whats the second extra offset for?

@Banshee: SHP Builder ignores the 2 extra offsets on load and saves blank ones. If any SHP relies on the "Loop" ability then SHP Builder would invalidate it when it saves.

Blade wrote:
I don't get why you are so invested in attaching the data to the end of the file when to any other application is just going to be extraneous junk at best, why not just keep it separate?

Keeping it separate is less convenient and "floods" a directory with additional files. I prefer clean solutions where possible.

Banshee wrote:
Stucuk, you should stop being stubborn for a while and listen to Blade's advise

I will always be stu born.

Banshee wrote:
Do not expect people in the future may not necessarily follow your standards, even if they happen to know and use your program.

I never do. Its also why i have stated from the beginning that everything will be --> optional <--.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Wed Aug 24, 2016 1:31 am    Post subject: Reply with quote

@Stucuk:
I think it is time to abandon the old names people have used in various projects. Through mine and Blades time working on reverse engineering these games and talking to ex-WW programmers, we pretty much know the names for each type of compression/codec/algo etc used in the games.

I have been spending most of my evening going through the Shapeset tool and we have a good idea where and how it writes shape files, even the RLE code it uses (we can provide C pesudo for you if you want to impliment a true compressor or decompressor Smile

Back to top
View user's profile Send private message
tomsons26lv
Pyro Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Wed Aug 24, 2016 1:32 am    Post subject: Reply with quote

stucuk wrote:
tomsons26lv wrote:
I tried adding extra data after the shp, as it stands i only tested it for a moment, in RA and RA2 it works, XCC says the shps are unknown, OpenRA crashes on game startup.

Are you adding it to TD/RA1 SHP's or TS/RA2 SHP's? XCC states the type as "shp (TS)" when extra data is saved to a TS/RA2 SHP. Based on what blade has said if its a TD SHP it will fail if you only add extra data (Though altering the eof offset may make it work. I still need to get round to coding the TD SHP writer)..

Thats why i mentioned RA and RA2, i tried both the formats and both the games as well as ORA
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
Banshee
Supreme Banshee


Joined: 15 Aug 2002
Location: Brazil

PostPosted: Wed Aug 24, 2016 1:57 am    Post subject: Reply with quote

stucuk wrote:
@Banshee: SHP Builder doesn't treat it like a flag.


But in this case, it doesn't make any difference at all.

stucuk wrote:
@Banshee: SHP Builder ignores the 2 extra offsets on load and saves blank ones. If any SHP relies on the "Loop" ability then SHP Builder would invalidate it when it saves.


I don't think it corrupts the file, because SHP Builder saves Format 80 frames only. So, it usually generates bigger files but it doesn't invalidate the files. Nonetheless, I'm interested to know what does a Loop Frame does and how does it work.


stucuk wrote:
Keeping it separate is less convenient and "floods" a directory with additional files. I prefer clean solutions where possible.

...

I never do. Its also why i have stated from the beginning that everything will be --> optional <--.


When someone chooses this option, the generated file might be invalid in other programs. I don't know how would the option be clear for any user. Many of them simply want to have the convenience of saving the file with the least bureaucracy necessary. So, I prefer to add one additional file in the directory than causing trouble by placing a optional feature that can break other programs, specially with users who doesn't pay attention to optional features.

Back to top
View user's profile Send private message Visit poster's website MSN Messenger
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Wed Aug 24, 2016 3:30 am    Post subject: Reply with quote

CCHyper wrote:
I have been spending most of my evening going through the Shapeset tool and we have a good idea where and how it writes shape files, even the RLE code it uses (we can provide C pesudo for you if you want to impliment a true compressor or decompressor Smile

With TD SHP's the decode80d which i "nicked"(Hard to nick when it has no license) from SHP Builder seems faulty (As its crashing on a certain TD SHP. I think its crashing with power.shp). I haven't looked at Blade's encoders.cpp and i doubt i will since its GPL (I don't want to make my project tied to "Virus" licenses. ZLib License is the best).

Any extra information/pesudo i can look at would be appreciated.

Banshee wrote:
But in this case, it doesn't make any difference at all.

Technically based on the official SHP's but if i saved a SHP with a flag of 2 and it had zero's in it then SHP Builder would not load it properly. If flag 2 is all the game needs to read it as RLE Zero then SHP Builder would be wrong. Also could be argued about any flag value with the 2nd bit enabled.

Banshee wrote:
I don't think it corrupts the file, because SHP Builder saves Format 80 frames only. So, it usually generates bigger files but it doesn't invalidate the files. Nonetheless, I'm interested to know what does a Loop Frame does and how does it work.

It won't corrupt it. But you lose the loop information.

I am guessing that when it reaches the end of the animation it can loop back to a frame to start again (As in not the first frame). So for example if you had an animation of radar being built and also had frames showing it rotate, you could specify the loop frame at the start of the radar rotating so that once it reached the end it would just show the radar rotating. Pure Speculation on my part as i have yet to find a SHP that actually has a loop.

Banshee wrote:
the generated file might be invalid in other programs

My code is a work in progress, its not a full release application and it can't currently do anything but view shp's and change the radar color. Its not Armageddon yet, there is still time to actually test stuff out. Hell its possible any code i write for saving TD's may be invalid when read by certain applications due to a bug in the code, but thats why stuff gets tested before any proper release.

If the TD Specification doesn't support it(Making the EOF offset point to the true EOF and not the end of the frame data) then it won't happen. There is a difference between the specification not supporting something and a Non-WW application assuming certain values have to be in certain places when they don't.

P.S Irrespective of what i do, since the project is open source even if i added in code to enable CABAL to take over your computers you could easily remove it.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
Bittah Commander
Defense Minister


Joined: 21 May 2003
Location: The Netherlands

PostPosted: Wed Aug 24, 2016 6:17 am    Post subject: Reply with quote

stucuk wrote:
Banshee wrote:
Nonetheless, I'm interested to know what does a Loop Frame does and how does it work.

I am guessing that when it reaches the end of the animation it can loop back to a frame to start again (As in not the first frame). So for example if you had an animation of radar being built and also had frames showing it rotate, you could specify the loop frame at the start of the radar rotating so that once it reached the end it would just show the radar rotating. Pure Speculation on my part as i have yet to find a SHP that actually has a loop.

I suspect that Art.ini turned this feature obsolete at some point, considering that you can already achieve exactly what you described via just Art.ini code (although you can't actually use buildup animations for this purpose).
_________________

Back to top
View user's profile Send private message Send e-mail MSN Messenger
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Wed Aug 24, 2016 7:42 am    Post subject: Reply with quote

The loop frame doesn't actually do anything and is unused by the game as far as I can tell. It looks like TD Shape is descended from the WSA format which is stored as deltas against the previous frame and so a loop frame makes it easy to generate the first frame from the last frame. Only a few shape file have this format in the original, a few radar images in Sole Survivor and all the shapes in wolapi.mix in the 3.0x+ patches for RA.

stucuk, consider this permission to port the encoders.cpp code under a zlib license to your tool. Properly supporting efficient compression is more important than wrangling over licenses. I doubt someone is going to pick it up and build a bunch of closed source products with it anyhow.

Back to top
View user's profile Send private message
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Wed Aug 24, 2016 9:54 am    Post subject: Reply with quote



Implemented the ISO building type. RA2 buildings with 8 frames will be assumed to be an ISO (Though it can't tell which one). The 4th frame will use the ISO palette.

SHP's can now only be classed as the following types: Unit, Building, Cameo, Animation, Building (IsoSnow), Building (IsoTemperate) and Building (IsoUrban). Types are used for the guess as to the palette as well as some rules like if it has half of its frames as shadow frames or not. If a SHP has a different extension (.sno, .tem, etc) it will use the extension to guess the palette over using the type.

Also implemented the ability to select a palette. The way the system works is that when a SHP is loaded it will guess what it is and use the Type and Game guesses to get the palette. If the user manually selects a palette and you then try to change the type or game it will not change the palette, only the "Rules".

Does anyone know if the first say 12 colours after 0 of a Unit/Building's palette are shadows? RA2 seems to use 1 and 12 for shadows, TD/RA using 4. I noticed that the RA2 units which use 12 have a darker shadow so i am wondering if its a scale where 1 is light shadow and 12 is dark or if they are just special cases.

Blade wrote:
stucuk, consider this permission to port the encoders.cpp code under a zlib license to your tool. Properly supporting efficient compression is more important than wrangling over licenses. I doubt someone is going to pick it up and build a bunch of closed source products with it anyhow.


Thanks. You never know, EA may want to use it for a browser game.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
Crimsonum
Seth


Joined: 14 Jul 2005
Location: The fine land of Finland

PostPosted: Wed Aug 24, 2016 1:38 pm    Post subject: Reply with quote

stucuk wrote:
Does anyone know if the first say 12 colours after 0 of a Unit/Building's palette are shadows? RA2 seems to use 1 and 12 for shadows, TD/RA using 4. I noticed that the RA2 units which use 12 have a darker shadow so i am wondering if its a scale where 1 is light shadow and 12 is dark or if they are just special cases.


No, in TS and RA2 the shadow is drawn by any colour other than #0 as long as its a seperate shadow frame, see E1 Elite's post below (TD/RA use a different palette altogether, one that's preserved as palette.pal in the later games, so I can't say for sure how it behaves). In normal frames, colours #1-#255 are fully opaque, #12 included. I believe they used the solid black colour for unit shadows in RA2 to make them more distinguishable.

However, it's worth to mention here that different colours of the palette are affected differently by the ambient map lighting, and interestingly, the unit type. Observe: http://www.tiberiumweb.org/forums/index.php?showtopic=39&view=findpost&p=36709

This actually reminds me of something I've always felt lacking in SHP Builder: A way to simulate different ambient lighting levels. Basically what such tool would do is allow the user to lighten/darken only those select colours that are affected ingame (see the above link). It could have a simple lighting slider or value input, and an additional tick box to determine the type of the unit (SHP with or without turret) if this is not already determined by the SHP class. This would be very useful for designing unit muzzle anims and to see if there are any stray special-case colours in the wrong place.

Unfortunatelly I don't know if the palette is similarly affected in RA2/YR.
_________________
Twisted Insurrection mapper & moderator

Quote:
When it comes to email HTML, all best practices from web development goes straight out the window.

Last edited by Crimsonum on Wed Aug 24, 2016 2:04 pm; edited 1 time in total

Back to top
View user's profile Send private message Visit poster's website
E1 Elite
Railgun Soldier


Joined: 28 May 2013

PostPosted: Wed Aug 24, 2016 1:58 pm    Post subject: Reply with quote

#1 if used will look blue, not semi-transparent in normal frames. There is no gradient for
shadows. Though #1 is the designated color for shadows in shadow frames, but game can
handle any color other than #0 as shadow in shadow frames.

Back to top
View user's profile Send private message
Crimsonum
Seth


Joined: 14 Jul 2005
Location: The fine land of Finland

PostPosted: Wed Aug 24, 2016 2:05 pm    Post subject: Reply with quote

Ah yes, I had forgotten about that. I'm too used to the custom of using only colour #1 for shadow frames. Edited my original post.
_________________
Twisted Insurrection mapper & moderator

Quote:
When it comes to email HTML, all best practices from web development goes straight out the window.

Back to top
View user's profile Send private message Visit poster's website
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Wed Aug 24, 2016 8:15 pm    Post subject: Reply with quote

Crimsonum wrote:
(TD/RA use a different palette altogether, one that's preserved as palette.pal in the later games, so I can't say for sure how it behaves).

TD/RA1 don't seem to have any SHP's with shadow frames. They use #4 for shadows.

Crimsonum wrote:
This actually reminds me of something I've always felt lacking in SHP Builder: A way to simulate different ambient lighting levels. Basically what such tool would do is allow the user to lighten/darken only those select colours that are affected ingame (see the above link). It could have a simple lighting slider or value input, and an additional tick box to determine the type of the unit (SHP with or without turret) if this is not already determined by the SHP class. This would be very useful for designing unit muzzle anims and to see if there are any stray special-case colours in the wrong place.


Given the way SHPToolz is coded this shouldn't be hard to implement. So basically just need to make it alter everything but the last 16 palette colours(From what i can tell a custom palette would simulate a turret shp).
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Wed Aug 24, 2016 10:34 pm    Post subject: Reply with quote



Implemented the "Ambient Lighting". I implemented it based on changing the Luminosity of the Palette colours. I don't know if this is the right approach (Though technically it should be the "proper" way to do it). When its set to 0 it will have no effect (Middle of the Trackbar). Currently with a range of -50 to 50.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
tomsons26lv
Pyro Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Thu Aug 25, 2016 1:47 am    Post subject: Reply with quote

Haven't checked in game but WW did use HSV for the palette manipulations so highly possible is accurate.

Can you add printing out the Zero? dword too to the debug option?
Apparently at some point ShapeSet had a SHP linking option, the code is still somewhat in the game and ShapeSet just unknown in what state.

I suspect the shps that have data there contain a WWCRC32(just like mixes) or some other of  many WWs hash methods checksum of the linked file filename.
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
E1 Elite
Railgun Soldier


Joined: 28 May 2013

PostPosted: Thu Aug 25, 2016 5:01 am    Post subject: Reply with quote

stucuk wrote:
I implemented it based on changing the Luminosity of the Palette colours.

That won't change the SHP as the palette index would be the same. It is like viewing the SHP
with another palette.

Back to top
View user's profile Send private message
stucuk
Geek


Joined: 27 Aug 2002

PostPosted: Thu Aug 25, 2016 5:32 am    Post subject: Reply with quote

Does anyone have any recent information on TMP files? I found some information about both RA and TS but Olaf's TS one doesn't go into much specifics (Like how to work out how many tiles are in the file).

- http://xhp.xwis.net/documents/TMP_TS_Format.html
- http://cnc.wikia.com/wiki/Red_Alert_File_Formats_Guide

tomsons26lv wrote:
Can you add printing out the Zero? dword too to the debug option?
Apparently at some point ShapeSet had a SHP linking option, the code is still somewhat in the game and ShapeSet just unknown in what state.filename.

I can't find any SHP's which are not either TD/RA1 or a TMP file which have a Zero as the first 2 bytes.

E1 Elite wrote:
That won't change the SHP as the palette index would be the same. It is like viewing the SHP
with another palette.

Unless i misunderstood Crimsonum, he seemed to want the ability view a SHP with different ambient lighting. SHP Builder has a lighten/darken tool for pixels.
_________________
Free Map Editor - Game Requirements - Stucuk.Net

Back to top
View user's profile Send private message Visit poster's website
E1 Elite
Railgun Soldier


Joined: 28 May 2013

PostPosted: Thu Aug 25, 2016 5:52 am    Post subject: Reply with quote

TMP format was lately discussed here - http://www.ppmforums.com/viewtopic.php?p=544214#544214

Back to top
View user's profile Send private message
tomsons26lv
Pyro Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Thu Aug 25, 2016 11:30 am    Post subject: Reply with quote

stucuk wrote:
Does anyone have any recent information on TMP files? I found some information about both RA and TS but Olaf's TS one doesn't go into much specifics (Like how to work out how many tiles are in the file).

- http://xhp.xwis.net/documents/TMP_TS_Format.html
- http://cnc.wikia.com/wiki/Red_Alert_File_Formats_Guide

tomsons26lv wrote:
Can you add printing out the Zero? dword too to the debug option?
Apparently at some point ShapeSet had a SHP linking option, the code is still somewhat in the game and ShapeSet just unknown in what state.filename.

I can't find any SHP's which are not either TD/RA1 or a TMP file which have a Zero as the first 2 bytes.


I posted WWs OFFICIAL header for the Icon format in the first page already :/
The TD Icon is just the header and the image data afik, haven't looked into it that much.

The Icon format in RA is a new version actually, the header is the same but now in addition to image data it contains OccupyList and OverlapList for each tile and some other bits of information. In TD the info was hardcoded into the exe.

Random fun fact, the Icon format has been around since 1988 with BattleTech: The Crescent Hawk's Inception being the first WW game to use it. That format version was largely IFF specification based.

Not really sure this tool should have Icon support frankly, maybe it should be a separate tool..

What are you talking about.. See the ts shp header i posted in the same post where the Icon(tmp) header is.
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Last edited by tomsons26lv on Thu Aug 25, 2016 2:21 pm; edited 3 times in total

Back to top
View user's profile Send private message Visit poster's website
Crimsonum
Seth


Joined: 14 Jul 2005
Location: The fine land of Finland

PostPosted: Thu Aug 25, 2016 1:44 pm    Post subject: Reply with quote

stucuk wrote:
E1 Elite wrote:
That won't change the SHP as the palette index would be the same. It is like viewing the SHP
with another palette.

Unless i misunderstood Crimsonum, he seemed to want the ability view a SHP with different ambient lighting. SHP Builder has a lighten/darken tool for pixels.


Correct, what you did is exactly what I wanted. Thanks!

I noticed a small error, though: The last colour in the palette (#255) is actually affected by the ambient lighting. So only colours from #240 to #254 should not be affected by the luminosity change.
_________________
Twisted Insurrection mapper & moderator

Quote:
When it comes to email HTML, all best practices from web development goes straight out the window.

Back to top
View user's profile Send private message Visit poster's website
tomsons26lv
Pyro Sniper


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Thu Aug 25, 2016 2:03 pm    Post subject: Reply with quote

Crimsonum wrote:
stucuk wrote:
E1 Elite wrote:
That won't change the SHP as the palette index would be the same. It is like viewing the SHP
with another palette.

Unless i misunderstood Crimsonum, he seemed to want the ability view a SHP with different ambient lighting. SHP Builder has a lighten/darken tool for pixels.


Correct, what you did is exactly what I wanted. Thanks!

I noticed a small error, though: The last colour in the palette (#255) is actually affected by the ambient lighting. So only colours from #240 to #254 should not be affected by the luminosity change.


Yea you should do some ini config file for pals to define the indexes that don't get affected by lighting, i highly doubt its just unit*.pals
_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Thu Aug 25, 2016 2:10 pm    Post subject: Reply with quote

Afaik it's really only the colors #240 to #254 in the unit*.pal's that stay in normal brightness, regardless of the maps Ambient setting.

For other pals it's either all colors get affected or none.
Thus only on the unit*.pal's you can have graphical issues with glowing pixel.

An ini might be only useful if OpenRA (or other engine) implements such a concept in a user definable way and thus this tool trying to stay compatible for such future projects.
_________________
SHP Artist of Twisted Insurrection. Main Nod building SHP creator.

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant unit, 1 Scrin unit, 1 GDI building

Tools
TMP Shop
C&C Executable Modifier

Back to top
View user's profile Send private message
Graion Dilach
Defense Minister


Joined: 22 Nov 2010
Location: Iszkaszentgyörgy, Hungary

PostPosted: Thu Aug 25, 2016 2:36 pm    Post subject: Reply with quote

In, OpenRA, any palette with AllowsModifiers: false stay in normal brightness, regardless of lighting. Changing this via mixing lit-unlit colors to a single palette probably won't happen because that harms renderer performance.
_________________



AS Discord server: https://discord.gg/7aM7Hm2

Back to top
View user's profile Send private message MSN Messenger
pchote
Cyborg Soldier


Joined: 06 Feb 2015

PostPosted: Thu Aug 25, 2016 3:27 pm    Post subject: Reply with quote

OpenRA has supported traits that modify only a subset of a palette since approximately forever. This is how the water / stealth / blinking light effects work.  The "global lighting" modifiers that change the whole palette equally is just a special case of that logic.

Back to top
View user's profile Send private message
E1 Elite
Railgun Soldier


Joined: 28 May 2013

PostPosted: Fri Aug 26, 2016 7:23 am    Post subject: Reply with quote

At first glance, I thought Crimsonum wanted more than just a view feature as a custom palette
would have given that info on the glowing pixels. If that same logic is available for isotem.pal
or non unit*.pal custom palettes, it would give incorrect info.

If you are looking for more viewing features, some ideas:
- tool could provide a mask on the palette to show which colors are used or not used for
current frame or in the first half of the frames or the whole of the SHP.
- or a selectable palette where user can click on colors to be pinked out to see if the image
is affected.
- animation could have a loop from-to frame numbers to view only a subset of frames looped
- tool could show the compression used alongside the frame counter.
- could allow the background scene to come from an image. It might help in screenshot or
even cameo making.

There could be more complex ones if you get into comparing 2 SHPs or frames.

Hope your palette dropdown extends for longer filenames like unttemmask1ra2bgremap.pal.
And I wouldn't want those palette colors to be saved along with the SHP file as half of the
palette colors would be masked pink color. But instead it would be for viewing with unittem.
As I understand the palette saving with SHP will be optional.

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Fri Aug 26, 2016 7:29 am    Post subject: Reply with quote

^^excellent ideas

As for the custom background image suggestion, it would be very useful if you could load multiple SHPs for one preview. So you don't have to go ingame to check how it turns out.
Just load the background image (screenshot of a ingame landscape), load the SHPs, set the loop frames for each SHP and click play for a nice preview.

Then a working gif exporter would be great (not this color ruining thing in SHP builder)
_________________
SHP Artist of Twisted Insurrection. Main Nod building SHP creator.

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant unit, 1 Scrin unit, 1 GDI building

Tools
TMP Shop
C&C Executable Modifier

Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic Page 2 of 3 [148 Posts] Goto page: Previous 1, 2, 3 Next
View previous topic :: View next topic
 Forum index » Featured Tools » OS SHP Builder
Jump to:  
Quick Reply
Username:


If you are visually impaired or cannot otherwise play the game below please contact the Administrator for help.


 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © phpBB Group

Wildcard SSL Certificates
[ Time: 0.7490s ][ Queries: 14 (0.0067s) ][ Debug on ]