KeithM At Home. KeithM At Home.on the web since 2010.not retro.just old-fashioned
The x3f-raw-format document details the compression schemes for the SD series camera, but excludes the DP series camera. The schemes used for the SD camera do however provide an insight into the DP series. Having decoded the Directory of a sample image and found the start of the raw image block at 00165B68 we start with the following data:
00165B60: 53 45 43 69 00 00 02 00 r.a.l... SECi.... 00165B70: 03 00 00 00 1E 00 00 00 80 0A 00 00 00 07 00 00 ........ Ç....... 00165B80: 30 E9 96 00 0Úû..... .....Ç.@
Interpreting this SECi block, we get:
4 bytes: 53 45 43 69 = SECi : Section Identifier 4 bytes: 00 00 02 00 = 2.0 : Version ( 2 bytes minor, 2 bytes major ) 4 bytes: 03 00 00 00 = d3 : image data type : 3 = other 4 bytes: 1E 00 00 00 = d30 : Data format : 30 = Unknown RAW image compression in Sigma DP1 camera ( and DP2 it seems ) 4 bytes: 80 0A 00 00 = d2688 : width of image ( or image columns ) 4 bytes: 00 07 00 00 = d1792 : height of image ( or image rows ) 4 bytes: 30 E9 96 00 = d9890096 : RowSize : value suggests block length...
On comparing a couple of files, the next few bytes always had the same values:
00165B80: 00 02 00 02 00 02 00 00 03 80 03 40 0Úû..... .....Ç.@ 00165B90: 03 20 04 C0 04 60 04 E0 04 70 04 F0 04 10 04 A0 . .+.`.Ó .p....á 00165BA0: 05 D0 06 D8 06 DC 00 00 .ð.Ï._.. .!1.¯¦2.
This block starts with four 16bit numbers:
00 02 = d512 00 02 = d512 00 02 = d512 00 00 = d0
which are the starting values for differential Huffman decoding for R, G and B. In all the DP2 files I examined, these values were always the same. I still read and use these values though when decoding the file. Next we have fourteen pairs of bytes, which are thirteen Huffman table entries followed by terminating zeros:
03 80 00000011 10000000 03 40 00000011 01000000 03 20 00000011 00100000 04 c0 00000100 11000000 04 60 00000100 01100000 04 e0 00000100 11100000 04 70 00000100 01110000 04 f0 00000100 11110000 04 10 00000100 00010000 04 a0 00000100 10100000 05 d0 00000101 11010000 06 d8 00000110 11011000 06 dc 00000110 11011100 00 00 00000000 00000000
As with the documented compression schemes, the first byte gives the number of bits and the second byte gives the bit pattern. The leaf values are inferred from the order of the table so the first entry is leafvalue = 0, the last entry is leafvalue = 12. Hence:
Bit pattern Leaf Value 100 0 010 1 001 2 1100 3 0110 4 1110 5 0111 6 1111 7 0001 8 1010 9 11010 10 110110 11 110111 12
We are now here in the image block:
00165BA0: 14 21 31 00 EE CC 32 00 .ð.Ï._.. .!1.¯¦2. 00165BB0: E9 FA 32 00 A0 06 9F F4 3A A2 14 F2 42 43 29 51 Ú·2.á.ƒ¶ :ó.=BC)Q 00165BC0: 62 C4 32 14 C6 29 64 92 2C 48 A7 42 49 24 86 52 b-2.ã)dÆ ,HºBI$åR
We next read three 32bit values:
h00312114 = d3219732 h0032ccee = d3329262 h0032fae9 = d3341033
These values are the number of bytes to be read for three blocks of data for red, green and blue respectively. The blocks however always start on a 16 byte boundary, so are zero padded. Looking at the start and end of the three blocks:
00165BB0: A0 06 9F F4 3A A2 14 F2 42 43 29 51 Ú·2.á.ƒ¶ :ó.=BC)Q 00165BC0: 62 C4 32 14 C6 29 64 92 2C 48 A7 42 49 24 86 52 b-2.ã)dÆ ,HºBI$åR
First byte is at h165BB4. So if we now read h312114 bytes, that takes us to h477CC7, which is here:
00477CC0: 75 22 49 24 8B 12 49 24 00 00 00 00 00 00 00 00 u"I$ï.I$ ........ 00477CD0: 00 00 00 00 D3 FF A7 FD 62 49 24 5C 89 8E 1D 48 ....Ë º² bI$\ëÄ.H 00477CE0: 8B 12 1E 11 62 49 0F 16 12 43 22 44 34 2C 99 E6 ï...bI.. .C"D4,Öµ
We now have 12 bytes of '00' which aligns us to a 16 byte boundary from where we started. Restart from h477CD4, and now read h32CCEE bytes, that takes us to h7A49C1, which is here:
007A49C0: 64 93 00 00 A0 06 9F F4 58 B0 B2 1D 48 91 62 48 dô..á.ƒ¶ X¦¦.HæbH 007A49D0: 78 B3 1C 58 8B 12 49 22 C5 31 52 48 B0 B2 45 88 x¦.Xï.I" +1RH¦¦Eê
We now have 2 bytes of '00' to get to the boundary. Restart at h7A49C4, read h32FAE9 bytes, which takes us to hAD44AC, which is here:
00AD44A0: 90 D0 A6 85 92 1A 16 49 16 22 C4 92 49 00 00 00 ÉðªàÆ..I ."-ÆI... 00AD44B0: 00 00 00 00 53 45 43 69 00 00 02 00 02 00 00 00 ....SECi ........
and we have 7 bytes to get us to the boundary, which is where the next SECi block starts. Phew !
In outline, the program flow to decode the image block is:
There is, however, a twist. With the SD compression schemes, the first value in a row is a decoded value and subsequent values are differential. The scheme used for the DP series is subtly different - which is where the ackowledgement to Dave Coffin's DCRaw comes in.
First, for SD cameras, the differential values are simply added to the previous value within the bit-length of the image ( 8 or 16bit ) - that is 8bit or 16bit addition. The scheme used for DP cameras closely follows that used for Nikon cameras. When reading bits for the differential value, the following code is used to retrieve the number of bits to be read, and then the value:
Private Function GetDiffIndirectValue() As Integer
Dim i As Integer
Dim bFlip As Boolean
Dim nNode, nNumBits, nValue As Integer
'start from top node in tree. It has no value ( -1 ), so we will move from here...
nNode = 0
nNumBits = HuffmanTree(nNode).nLeafValue
While (nNumBits = -1) 'while we're sitting on a leaf that has no value ( -1 )...
nNode = HuffmanTree(nNode).nBranches(GetBit()) 'traverse tree to next node down the indicated branch...
If (nNode = -1) Then 'not a valid branch - error reading data so bail out...
szGlobalErrMsg &= "GetDiffIndirectValue: Dead-end Forwarder..." & vbCrLf
Return (Integer.MinValue)
End If
nNumBits = HuffmanTree(nNode).nLeafValue 'get the leaf value at this node...
End While
If (nNumBits = 0) Then Return (0)
nValue = GetBit()
bFlip = Not CBool(nValue) 'if first bit is '0', then we need to flip value...
For i = 1 To nNumBits - 1
nValue = GetBit() + (nValue << 1)
Next
If (bFlip) Then nValue -= (1 << nNumBits) - 1
Return (nValue)
End Function
And for the final turn of the twist... with the SD cameras, the previous decoded value used is always from the previous pixel. Not so the DP scheme. Again, this code has it's roots in DCRaw. Having taken the easy option of declaring a couple of public variables:
Public vpred(3), hpred(1) As Integer
before decoding each block of R, G and B, load the respective starting value into an array:
'load vpred array with respective predictor value..
For i = 0 To 3
vpred(i) = PredictorValue
Next i
where PredictorValue is the required R, G or B starting value for the colour you are decoding. The loop for decoding the image values is:
For y = 0 To nImageHeight - 1
For x = 0 To nImageWidth - 1
nValue = GetDiffIndirectValue()
If (nValue = Integer.MinValue) Then
szGlobalErrMsg &= "DecodeSigma0330Image: Decode error..." & vbCrLf
GoTo ohooh
End If
If (x < 2) Then
i = 2 * (y And 1) + (x And 1)
vpred(i) += nValue
hpred(x) = vpred(i)
Else
hpred(x And 1) += nValue
End If
ImageOut(x, y).nR = (hpred(x And 1))
Next x
Next y
this being the particular loop for Red.
Since this code was originally written to decode compressed images from Bayer pattern sensors, this kind of implies that the compression engine used in the True II processor originally started life in a Bayer camera !!
Hopefully I've entered all the above data correctly. My apologies if I've typo'd anything - I don't usually spot spelling mistakes until the precise moment after hitting the Send button.
Page Last Updated: 28th February 2010, Maintained by KeithM.