Posted: Sun Jun 14, 2015 9:23 am Post subject:
Reading and parsing through vxl files...
I am wondering if anyone can explain how to read and parse through vxl files...
I have been looking at Eol/DMZ VXL_Format... And its really not making since...
I do see its little endian but that doesn't seem to help much...
I do not know much about binary or endian file types... And looking on-line can be a mess as there is so many different types that I have no idea were to start.
How would I go about getting the data to ascii?
for example I am trying to retrieve the vxls palette color index,normal value and i guess vxls them-self...
Also how are voxels stored... an 2d array?
[x,x,x,x]
[x,0,0,x]
[x,0,0,x]
[x,x,x,x]
x's values are just empty/none... and 0's are i guess used/on. _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
It starts at procedure TVoxel.LoadFromFile(FName: string);
It reads the header:
Code:
TVoxelHeader = packed record
FileType: packed array[1..16] of Char; // always "Voxel Animation"
NumPalettes, // always 1
NumSections,
NumSections2,
BodySize: LongInt; // as above
StartPaletteRemap,
EndPaletteRemap: Byte; // colour indexes
PaletteData: packed array[1..256*3] of Byte; // never used
end;
Then it reads all headers, bodies and tailers (as explained in these DMZ/Eol, etc articles. VXLSE III calculates their position to read section per section.
Then, it reads each section (all at the same file ):
Look for function TVoxelSection.LoadFromFile(var F: File; HeadOfs, BodyOfs, TailOfs : Integer): EError;
It basically reads the header, body and tailer. Then, it reads the data, which is an array data with some kind of compression, since I don't think it stores the blank voxels explicitly. You'll need to adapt that code for your programming language. It's not that hard at all. QUICK_EDIT
Joined: 22 Nov 2010 Location: Iszkaszentgyorgy, Hungary
Posted: Sun Jun 14, 2015 5:36 pm Post subject:
You can find OpenRA's VXL renderer code in https://github.com/OpenRA/OpenRA/tree/bleed/OpenRA.Game/Graphics - tho don't ask me how it works. _________________ "If you didn't get angry and mad and frustrated, that means you don't care about the end result, and are doing something wrong." - Greg Kroah-Hartman
=======================
Past C&C projects: Attacque Supérior (2010-2019); Valiant Shades (2019-2021)
=======================
WeiDU mods: Random Graion Tweaks | Graion's Soundsets
Maintainance: Extra Expanded Enhanced Encounters! | BGEESpawn
Contributions: EE Fixpack | Enhanced Edition Trilogy | DSotSC (Trilogy) | UB_IWD | SotSC & a lot more... QUICK_EDIT
$contents = fread($handle, 15);//anything more than this and it reads to much
print_r( $contents );//this will display "Voxel Animation"
echo "<br>";
*** This is were I get lost... Should I only take in 1 byte... or 32 bits for a long?
** All I know is I should be getting "1"....
$contents = fread($handle, 1);
print_r( bin2hex($contents) ); ** And I think this needs to be packed... I do not understand that stuff either... [url]http://php.net/manual/en/function.pack.php[/url]
** if I read 1 byte I get 00
** if I read 32 I get 00010000000100000001000000dc080000101f00000000000000000000000000
fclose($handle);
?>
The vxl header looks some thing like this...
struct vxl_header
{
char filetype[16]; /* ASCIIZ string - "Voxel Animation" */
long unknown; /* Always 1 - number of animation frames? */
long n_limbs; /* Number of limb headers/bodies/tailers */
long n_limbs2; /* Always the same as n_limbs */
long bodysize; /* Total size in bytes of all limb bodies */
short int unknown2; /* Always 0x1f10 - ID or end of header code? */
char palette[256][3]; /* 256 colour palette for the voxel in RGB format */
};
I have also upload the cube vxl I am testing this on if any one can is willing to help me understand this...
One thing I've done using TCL to handle binary files (or binary sockets) is to read clumps at a time, something larger than the file's possible header, then parse that in memory... to start. If you know the next piece of info is 8 bytes, process the next 8 bytes from your clump, and of course do some checks to ensure it conforms to what you expect.
Use fread(expected length), or fread _more_ but only parse X bytes, is it not working?
I was just thinking maybe PHP is deciding you hit EOF when you haven't, I remember having a similar problem when I wrote my TCL wget-like script for sucking jpegs and other files (used in conjunction with another script I wrote that generated the grab list). _________________ http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai QUICK_EDIT
... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) Last edited by MadHQ on Fri Jun 19, 2015 12:36 pm; edited 1 time in total QUICK_EDIT
I was just thinking maybe PHP is deciding you hit EOF when you haven't, I remember having a similar problem when I wrote my TCL wget-like script for sucking jpegs and other files (used in conjunction with another script I wrote that generated the grab list).
I dont think I want to read the whole file... I really just want this file in a human read able form... I know this sounds crazy... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Looks like a double-word, is 2 "words" put together, I believe the number you are reading is correct, think of it like little-endian vs big-endian but in blocks of 2 bytes. So like 5,6,7,8,1,2,3,4 in significance...
The vxl format description by Eol/DZM says it little-endian...
I do not know enough about binary files/data... I have tried googling the hell out of this topic, with no luck... There is just way to much out there to narrow down a search that makes since...
Are you saying I have to swap the bits(or bytes)? _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
type
TVoxelHeader = packed record
FileType: packed array[1..16] of Char; // always "Voxel Animation"
NumPalettes, // always 1
NumSections,
NumSections2,
BodySize: LongInt; // as above
StartPaletteRemap,
EndPaletteRemap: Byte; // colour indexes
PaletteData: packed array[1..256*3] of Byte; // never used
end;
For the palette and section counts you can probably discard the high bits and do a hex conversion on the lowers, then for the body size treat them as 2 hex conversion calculations. Multiply the upper bits by 65536 and add to the lowers. _________________ http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai Last edited by G-E on Fri Jun 19, 2015 1:02 pm; edited 1 time in total QUICK_EDIT
I guess I have FileType, as that php code I have can read it just fine. (One of the two none binary data in these files... LOL)
Now I am trying to get NumPalettes witch should be "1" I guess
I know I am really far... I cant get past the binary knowledge step...
Quote:
For the palette and section counts you can probably discard the high bits and do a hex conversion on the lowers, then for the body size treat them as 2 hex conversion calculations.
That gives you the actual number, but like I say for the palettes and section counts, you can ignore bytes 2-4, just use the first one, there's no way anyone is going to have 257 voxel sections hahah _________________ http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai QUICK_EDIT
set num [expr (0x$b4 * 16777216) + (0x$b3 * 65536) + (0x$b2 * 256) + 0x$b1]
What is the point of this? And I what is it doing, Converting base, if so what base is that, and how did you find that out? _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Because we know it's likely 1, but for sure it's less than 257, so the number will never exceed the first byte. Remember least significant byte first, 0-255.
MadHQ wrote:
Quote:
set num [expr (0x$b4 * 16777216) + (0x$b3 * 65536) + (0x$b2 * 256) + 0x$b1]
What is the point of this? And I what is it doing, Converting base, if so what base is that, and how did you find that out?
I just broke it apart so you could see how each byte adds up to a total number, there are more compact ways of doing that of course.
In PHP you need to use pack/unpack and bin2hex/hexdec to reformat the data, and you can tell it to unpack little-endian specifically, skipping my reverse assembly thing QUICK_EDIT
Thats what I am at as of right now... I am just trying to recreate the size section for right now... I am trying to base it off you code with the whole multiply thing... As I do not understand the pack/unpack thing in PHP... It just doesn't seem like its working...
So whats wrong with the last section I have....?
Quote:
Voxel Animation
1
1
1
3690987520
This is the output I am getting...
The size seems way to big... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Current reads:
struct vxl_header
struct vxl_limb_header's
does any one know if the PalleteData section is empty? I tried getting numbers out of it with no luck... always seemed to be "0"... So I just skipped 768 bytes... Seems like its fine as its able to read the limb headers...
And I am still not sure my BodySize is correct... It should be... As I am using the same function as the NumSections... and that looks good. _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Looks like you have to work backwards using the body size, minus the multiple of tailers to find where they start. So if I read that right, a 3 section voxel has:
The first 3 lines seem fine... But the floats do not look right... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Whats bothering me is this "Det: Single;" This is another floating point...
and it looks like there array is [3][4]....
In DMZ/Eol There is no Det... as showen bellow... They do not even have that extra float... Witch one is correct?
Code:
struct vxl_section_tailer
{
long span_start_off; /* Offset into body section to span start list */
long span_end_off; /* Offset into body section to span end list */
long span_data_off; /* Offset into body section to span data */
float transform[4][4]; /* Inverse(?) transformation matrix */
float scale[3]; /* Scaling vector for the image */
char xsize; /* Width of the voxel limb */
char ysize; /* Breadth of the voxel limb */
char zsize; /* Height of the voxel limb */
char unknown; /* Always 2 - unknown */
};
And I am still not sure on how to deal with floats... I keep trying... and am haveing no luck... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Joined: 22 Nov 2010 Location: Iszkaszentgyorgy, Hungary
Posted: Sat Jun 20, 2015 10:31 am Post subject:
Question: why php? _________________ "If you didn't get angry and mad and frustrated, that means you don't care about the end result, and are doing something wrong." - Greg Kroah-Hartman
=======================
Past C&C projects: Attacque Supérior (2010-2019); Valiant Shades (2019-2021)
=======================
WeiDU mods: Random Graion Tweaks | Graion's Soundsets
Maintainance: Extra Expanded Enhanced Encounters! | BGEESpawn
Contributions: EE Fixpack | Enhanced Edition Trilogy | DSotSC (Trilogy) | UB_IWD | SotSC & a lot more... QUICK_EDIT
Okay if you go to this link it will parse through the vxl data...
Now what I am totally blown away by is 30 bytes off...
The files SpanDataOfs = 800
but!!!
the real vxl data should start at 830...
header = 802 bytes
limb header = 28 bytes
add those two up and you get 830...
Now I am not sure whats really meant to be in span_start[n] & span_end[n], but I should be able to just read through 100 long(4 bytes each) for both span_start[n] & span_end[n], as my X and Y (10*10)...
But there is a 30 bytes still floating around...
I was able to look in OS vxl edit, and match vxl pal index and normal index with data with from the php script to find out theirs a 30 bytes problem...
802 is the fixed header size with the 768 palette bytes, before the section headers, and you know each tailer is 92 bytes. This means your actual section data is between start+830 and end-92 in your case...
Or use unpack like I suggested earlier, I just looked it up and you can indeed parse 32bit integers and floats with it directly into an array... perhaps you can use it to unpack whole sequences using a larger read? _________________ http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai QUICK_EDIT
I thought it didn't work but it was just that i was not paying attention to the array index...
unpack returns an array, index "0" is used for some thing else... and that's index i was trying to pull the float out of... as it turns out the returning float i wanted was in index "1"...
Silly me...
And now I feel as though I have all the data out of a vxl file that I need... as shown in link below! I improved my span_data read code... Got it working right.
For some reason they return as "nan" (I guess not a real number...)
Not to sure what to make out of that... Any one have any thoughts?
Anyways!!! I can be any more happy I think I have every thing I need to start working on the other part of this little project of mine.
Time to turn read vxl data into point cloud data... This will be fun! The good news is that I see the pattern of how the vxl span_dat is stored and how it represents a ... tank I guess.
Wish me luck!
AH and thanks a bunch G-E! you have really helped me allot with this! _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
But now my next problem is that voxel colors do not seem to make since... The pattern of colors does not look right...
And the colors are not right... I am just loading the unittem.pal... That should be okay... Though I did look in the vxl file and it does store rbg color values.... >_< I did try that with no luck...
Unless any one has any ideas on the vxl colors I guess i will just have to keep playing around... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
I'm sure Banshee could answer this all in like 2.3sec, I've had to learn the voxel format along with you to help haha...
Here's a clue about the packed data, VXLSE is doing bit shifts to find the right numbers:
Code:
function TVoxelSection.PackVoxel(Unpacked: TVoxelUnpacked): TVoxelPacked;
begin
Result := Unpacked.Flags shl 16;
if Unpacked.Used then
Result := Result or $100 or Unpacked.Colour
else
Result := Result or Unpacked.Colour;
Result := Result shl 8;
Result := Result or Unpacked.Normal;
end;
procedure TVoxelSection.UnpackVoxel(PackedVoxel: TVoxelPacked; var dest: TVoxelUnpacked);
begin
dest.Normal := (PackedVoxel and $000000FF);
dest.Colour := (PackedVoxel and $0000FF00) shr 8;
dest.Used := (PackedVoxel and $00010000) > 0;
dest.Flags := (PackedVoxel and $FF000000) shr 24;
end;
Now that that's established, if you can make this in an iframe in the context of a larger page that shows some numerical data about the model, that would be cool... still not sure what use it would be to anyone besides curiosity ? _________________ http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai QUICK_EDIT
Now that that's established, if you can make this in an iframe in the context of a larger page that shows some numerical data about the model, that would be cool... still not sure what use it would be to anyone besides curiosity ?
Do you want to see the data?
My plan with this was to try and make some kind of FLH locator.
Some thing else I need to get sorted through is the hva files, so I can get locations and possibly animations...
I have an attempt in the works right now... No to much luck so far...
This is as far as im at... Im just not sure on the file structure at all... It looks like I got the first parts right... but there's just to much data not used...
Also Known As: banshee_revora (Steam) Joined: 15 Aug 2002 Location: Brazil
Posted: Mon Jun 22, 2015 6:31 pm Post subject:
Just to answer a question from a PM from MadHQ here:
The Det from VXLSE III's is the Scale (It originally mistakenly stood for Determinant). In Voxel Section Editor III, you can view it in Section -> Voxel Properties -> Misc. It's the Scale setting there and it is usually expected to be 1/12 {0,0833333358168602} QUICK_EDIT
It seems i can find the transformation as long as there is no frames in hva... But I am not sure how to use the numbers as of right now... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Any chance I could use this to display units on the D-day wiki once you are done? I'm using Gif anims atm but this would be even better! I'm sure I'm not the only one that could make use of it too. _________________
Nice work, the ground is a nice addition, but no more zoom?
Ya I have temporary disabled zoom/pan un-tell I get the scale/sizes fully finished.
Mig Eater wrote:
Any chance I could use this to display units on the D-day wiki once you are done? I'm using Gif anims atm but this would be even better! I'm sure I'm not the only one that could make use of it too.
You know that would be pretty cool! I will see if I can come up with some thing like that.
But I really have a lot more to do and a few obstacles to overcome, before I can make any confirmations on that...
One of the obstacles is the normal's... It seems as though the way I am doing this is not going to allow for them...
IE... I am using a pointcloud to render these, and the points do not interact with lights...
My idea now is to just try and use the normal's as a color multiplayer... Downside with that is once is been generated it says like that... Not very normal-ish... but it could be okay...
BTW is there such a thing as a normal pallet in RA2/YR... I cannot seem to find it... Is is in voxel.vpl (I think that's whats its called...), It just that I remember seeing that grey scale pallet in the old vxl edit... I just cant find it in any mix's... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Yes voxels.vpl controls what resultant colour lightness "step" is... in your case I would just do it as a base add/subtract to the RGB value... add/sub 10-20 from whatever each byte of the 24bit value is (if they move together, tone stays the same until one hits 0 or 255).
You can also skip much of the work by generating a dot map of which pixels need highlighting at each facing (based on normals), but posterize it do say 4-6 levels only. So say you were looking at it from north, you would lump NNE/N/NNW into one level, NE/NW into the next level, and so on.
If you keep the camera angle the same, and only allow turning and zoom, this should be a fairly efficient way of calculating. There's only 256 possible normals, and you know that any given radial facing would have the same number of normals in your step groups, you'd only need a lookup table for each facing to decide which level it falls into. And while you could rotate it freely, you could set the radial steps for normals colouring to something like 15 degrees to reduce the number of lookup tables.
The lookup tables can be simple searchable arrays, one array for each facing... so in this example, 24 arrays, with 6 lightness/darkness levels, each level being a large bump to the colour. _________________ http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai QUICK_EDIT
Shouldn't each section have 256 different colors (with there own RGB value?)
Or am I looking at this wrong... is each normal color (x,x,x) x = one byte?
or is it ((x),(x+1),(x+2)) x = one byte.... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
Now its time to tweek in the normal's values so they do not look so bad...
I had to multiply the normal value by i think 0.15 as they keep bleaching to white... _________________ MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.) QUICK_EDIT
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You can download files in this forum