• Trion bitstream compression test

    From John Larkin@3:633/280.2 to All on Sun Dec 11 14:13:24 2022
    Reply-To: xx@yy.com


    Suppose we read in an FPGA bitstream file as 16-bit words, and want to
    output a compressed file, to save room in a small flash chip.

    If an input word is 0000, output a single 0 bit to the compressed
    file.

    If a word is nonzero, output a 1 followed by that 16-bit word.

    The decompressor will be very simple.

    I wrote a little PowerBasic program to test the compression, the input
    being an actual Efinix T20 bit stream file.

    https://www.dropbox.com/s/dkgvdsz2wytvgwm/Trion_Compression.jpg?raw=1


    --- MBSE BBS v1.0.8 (Linux-x86_64)
    * Origin: Highland Tech (3:633/280.2@fidonet)
  • From Dimiter_Popoff@3:633/280.2 to All on Sun Dec 11 16:16:41 2022
    On 12/11/2022 5:13, John Larkin wrote:

    Suppose we read in an FPGA bitstream file as 16-bit words, and want to
    output a compressed file, to save room in a small flash chip.

    If an input word is 0000, output a single 0 bit to the compressed
    file.

    If a word is nonzero, output a 1 followed by that 16-bit word.

    The decompressor will be very simple.

    I wrote a little PowerBasic program to test the compression, the input
    being an actual Efinix T20 bit stream file.

    https://www.dropbox.com/s/dkgvdsz2wytvgwm/Trion_Compression.jpg?raw=1


    If you get the results you need using it why not.
    The more trivial RLE is not too resource hungry and easy to
    implement, IIRC (have done it last over a decade ago, our VNC (RFB)
    server does it all the time during framebuffer transfers).

    --- MBSE BBS v1.0.8 (Linux-x86_64)
    * Origin: TGI (3:633/280.2@fidonet)
  • From Martin Brown@3:633/280.2 to All on Sun Dec 11 22:05:57 2022
    On 11/12/2022 03:13, John Larkin wrote:

    Suppose we read in an FPGA bitstream file as 16-bit words, and want to
    output a compressed file, to save room in a small flash chip.

    If an input word is 0000, output a single 0 bit to the compressed
    file.

    If a word is nonzero, output a 1 followed by that 16-bit word.

    The decompressor will be very simple.

    I wrote a little PowerBasic program to test the compression, the input
    being an actual Efinix T20 bit stream file.

    https://www.dropbox.com/s/dkgvdsz2wytvgwm/Trion_Compression.jpg?raw=1

    Well you *could* do that but if the sample file you posted a while back
    is any guide to their contents you might want to investigate the run
    length N of consecutive zero bits that gives maximum compression.

    My instinct is that it may be somewhere near 120000.
    Since there are entire blocks of 15k zero bytes in the test file.

    0 bit counts with N = 5,6,7, 8, 16, 32, 64 ... might also be worth investigating and stop when it no longer yields additional compression.

    Your sample file was 70% 00000000 and 20% with a single bit set.
    Its natural word length appears to be the 8 bit byte.

    My preference would be to encode it as 8 bit words and use the symbols
    which never occur or occur just once in the entire file to represent the following (not necessarily in this order).

    15k run of zeroes
    00
    000
    0000
    00000000

    and <token>,<token> for the token itself

    --
    Regards,
    Martin Brown


    --- MBSE BBS v1.0.8 (Linux-x86_64)
    * Origin: Aioe.org NNTP Server (3:633/280.2@fidonet)
  • From John Larkin@3:633/280.2 to All on Mon Dec 12 02:41:57 2022
    Reply-To: xx@yy.com

    On Sun, 11 Dec 2022 11:05:57 +0000, Martin Brown
    <'''newspam'''@nonad.co.uk> wrote:

    On 11/12/2022 03:13, John Larkin wrote:

    Suppose we read in an FPGA bitstream file as 16-bit words, and want to
    output a compressed file, to save room in a small flash chip.

    If an input word is 0000, output a single 0 bit to the compressed
    file.

    If a word is nonzero, output a 1 followed by that 16-bit word.

    The decompressor will be very simple.

    I wrote a little PowerBasic program to test the compression, the input
    being an actual Efinix T20 bit stream file.

    https://www.dropbox.com/s/dkgvdsz2wytvgwm/Trion_Compression.jpg?raw=1

    Well you *could* do that but if the sample file you posted a while back
    is any guide to their contents you might want to investigate the run
    length N of consecutive zero bits that gives maximum compression.

    My instinct is that it may be somewhere near 120000.
    Since there are entire blocks of 15k zero bytes in the test file.

    0 bit counts with N = 5,6,7, 8, 16, 32, 64 ... might also be worth >investigating and stop when it no longer yields additional compression.

    Your sample file was 70% 00000000 and 20% with a single bit set.
    Its natural word length appears to be the 8 bit byte.

    My preference would be to encode it as 8 bit words and use the symbols
    which never occur or occur just once in the entire file to represent the >following (not necessarily in this order).

    15k run of zeroes
    00
    000
    0000
    00000000

    and <token>,<token> for the token itself

    I did consider the chunk size N at 32 bits.

    If N=16, the compression gain for data=0 words is 16:1, and the
    penalty on top of nonzero words is 6%. That's not bad for a simple
    decoder.

    0.4 file size ratio is pretty good. I like simple and done. I can
    explain this to a smart intern in minutes.

    I'll run my little Basic program on some more complex FPGA bit
    streams, as soon as we make some. The current project has a serial
    flash chip per FPGA so doesn't need compression, but a new Pi-based
    product line might benefit from compression. I expect the FPGA designs
    to be fairly simple or absurdly simple. We will use the T20 chips
    because we could get them and bought a bucket full for $10 each.

    Clocked slow and not working hard, no PLLs, the 1.25 volt core power
    is milliamps. A tiny linear regulator from +5 will work fine.

    My guys say that compared to the big-guy FPGAs, the efinix design
    suite looks like garage-level engineering. That's a feature, not a
    defect. A lot of the suite is in plain-sight Python and there's no
    FlexLM horrors to fight.


    --- MBSE BBS v1.0.8 (Linux-x86_64)
    * Origin: Highland Tech (3:633/280.2@fidonet)
  • From Dimiter_Popoff@3:633/280.2 to All on Mon Dec 12 03:16:41 2022
    On 12/11/2022 17:41, John Larkin wrote:
    On Sun, 11 Dec 2022 11:05:57 +0000, Martin Brown
    <'''newspam'''@nonad.co.uk> wrote:

    On 11/12/2022 03:13, John Larkin wrote:

    Suppose we read in an FPGA bitstream file as 16-bit words, and want to
    output a compressed file, to save room in a small flash chip.

    If an input word is 0000, output a single 0 bit to the compressed
    file.

    If a word is nonzero, output a 1 followed by that 16-bit word.

    The decompressor will be very simple.

    I wrote a little PowerBasic program to test the compression, the input
    being an actual Efinix T20 bit stream file.

    https://www.dropbox.com/s/dkgvdsz2wytvgwm/Trion_Compression.jpg?raw=1

    Well you *could* do that but if the sample file you posted a while back
    is any guide to their contents you might want to investigate the run
    length N of consecutive zero bits that gives maximum compression.

    My instinct is that it may be somewhere near 120000.
    Since there are entire blocks of 15k zero bytes in the test file.

    0 bit counts with N = 5,6,7, 8, 16, 32, 64 ... might also be worth
    investigating and stop when it no longer yields additional compression.

    Your sample file was 70% 00000000 and 20% with a single bit set.
    Its natural word length appears to be the 8 bit byte.

    My preference would be to encode it as 8 bit words and use the symbols
    which never occur or occur just once in the entire file to represent the
    following (not necessarily in this order).

    15k run of zeroes
    00
    000
    0000
    00000000

    and <token>,<token> for the token itself

    I did consider the chunk size N at 32 bits.

    If N=16, the compression gain for data=0 words is 16:1, and the
    penalty on top of nonzero words is 6%. That's not bad for a simple
    decoder.

    0.4 file size ratio is pretty good. I like simple and done. I can
    explain this to a smart intern in minutes.

    I'll run my little Basic program on some more complex FPGA bit
    streams, as soon as we make some. The current project has a serial
    flash chip per FPGA so doesn't need compression, but a new Pi-based
    product line might benefit from compression. I expect the FPGA designs
    to be fairly simple or absurdly simple. We will use the T20 chips
    because we could get them and bought a bucket full for $10 each.

    Clocked slow and not working hard, no PLLs, the 1.25 volt core power
    is milliamps. A tiny linear regulator from +5 will work fine.

    My guys say that compared to the big-guy FPGAs, the efinix design
    suite looks like garage-level engineering. That's a feature, not a
    defect. A lot of the suite is in plain-sight Python and there's no
    FlexLM horrors to fight.


    I think you worry a bit too much about decompression complexity.
    Decompression is much simpler/less computationally intensive than
    compression, even if you do something like LZW. Probably you won't
    need anything like that of course but if you do it won't be a killer.

    --- MBSE BBS v1.0.8 (Linux-x86_64)
    * Origin: TGI (3:633/280.2@fidonet)