Tag Archives: GOES

GOES Satellite Decoding, Part 3

The last post left off with the demodulated data separated into the individual Coded Virtual Channel Data Units (CVCDU). This post will detail the steps necessary to decode the CVCDUs into a satellite image. A great resource for the LRIT specifications can be found here. I’ve collected some of the figures from this report to create an overall flow chart of the reception and decoding process. The figure below shows how the data is processed from the moment it reaches the antenna to the final LRIT file. There is some additional processing required to decompress the image, but this gives a nice overview of the required steps.

LRIT GOES Protocol

Concatenation of Figures 1 and 3 from NOAA LRIT Specifications

The CVCDU contains three separate parts, the header, the data payload, and the error correction symbols. The packet uses a Reed-Solomon (RS) error correction scheme (applied to both the header and the data payload) to help eliminate errors. This error correction scheme leaves the original data unmodified, so it isn’t absolutely necessary to implement this error correction. The attached m-files use RS data to check for and correct any bit errors. The SNR of the SETI data is high enough that in all the processing, I haven’t detected any bit errors, so if you are processing these data sets, you could safely ignore this step. I wanted to make sure that my demodulation scheme could tolerate lower SNR data sets so I’ve kept this step in the decoder.

The structure of the CVCDU header is shown below, again taken from the NOAA LRIT specification document.

GOES VCDU Header

The two most important fields in this packet are the Virtual Channel ID (VC ID) and the VCDU Counter. The VC ID identifies the type of packet being transmitted. An ID of 63 represents a fill packet, which contains no information. The ID for various images from different spacecraft will be different. Much of what I know about the VC ID I’ve learned just by observing the VC ID for different data packets. If anyone knows of a place which details the different VC IDs please let me know. One example VC ID for GOES-13 is 42, this represents a packet which can contain imaging data. The other field of note is the VCDU counter, this is incremented with every packet and is useful in determining if you’ve dropped any packets.

The next step is to isolate the data field from the CVCDU and form the multiplexing data protocol units (M_PDU) packet. This packet is relatively uncomplicated, containing a two field, 16 bit header and a data field. The spare field is unimplemented and can be safely ignored, the 11 bit first header field points to the beginning of the next CCSDS path protocol data units (CP_PDU). If the packet does not contain a new packet, this field will be set to all ones, which in decimal is 2047.

At the CP_PDU level, things begin to get a little more difficult since the fields of this packet can span multiple M_PDU packets. The CP_PDU can be split at any point so the decoder must be prepared to handle this. The very first decoder I implemented assumed that part of the CP_PDU packet would always be present between the start of the CP_PDU packet and the end of the M_PDU packet. This resulted in the following picture.

GOES Earth Packet Errors

The assumption I made about the certain fields always being present, caused packets to be dropped when those fields were not present, resulting in the horizontal lines.

The CP_PDU is broken down into 4 main sections, with some sub-sections. These are the Packet Identification, Sequence Control, Packet Length, and Data sections. Each of these sections has information critical to the decoding process.

The Packet Identification section contains the application ID field. This field specifies the type of data contained in the CP_PDU. Each imaging set contains a different APID to specify its contents. The image above is spread across 4 APIDs, each APID delivers the imaging data for a few hundred rows of the image.I believe the North America image was made up of APIDs: 1344, 1364, 1365, 1371. These contained the visible light images. There are additional data packets which contain imaging data in other wavelengths.

The Sequence Control section contains two very important fields. The first field is the sequence flag which specifies if this packet is the first, middle, or last packet in this APID sequence. A sequence flag of 1 identifies the first packet in the series. A sequence flag of 0 specifies a middle packet, and a sequence flag of 2 identifies the final packet in the series. The sequence counter is used to keep track of the packets and ensure nothing is missed.

The first packet contains a wealth of information on the transmitted data. This information is contained in a series of headers. The details of these headers can be found here. The number and type of headers included is dependent on the type of data transmitted. Imaging data is supposed to include the following headers: Primary, Image Structure, Annotation, NOAA Specific, Header Structure Record. In at least some of the imaging data, the Annotation records appeared to be missing. This didn’t, however, affect the overall decoding process.

I’ll use APID 1366 as an example of the type of data contained in these headers since all the packets for this image are contained in the data I’ve posted. I’ve appended the tables found NOAA LRIT Receiver Specifications pdf to include not only the descriptions but also the contents of the fields for this APID.

The first record is the primary header which is always included in the transmission. This gives the overall length of the header file, as well as the total size of the LRIT file being transmitted. This file size represents the uncompressed file size. The image rows are compressed before transmission, so the actual received data will be less than the value given here.

GOES Primary Header APID 1366

Primary Record, Appended Table 5

The next record is the Image Structure Record which contains details about the format of the image transmission. The values recorded here are used for the decompression after the reception of the entire image. The compression flag value of 1 indicates that RICE lossless compression is used on the image data. All the images I’ve seen have only used a single byte per pixel.

GOES Image Structure Record APID 1366

Image Structure Record, Appended Table 6

The navigation record includes geographical information about the image data. I don’t actually use any of the data contained in this record.

GOES Navigation APID 1366

Navigation Record, Appended Table 7

Like the previous record, the NOAA Mission Specific record contains information that I’m not currently using in the decoding process.

GOES NOAA Record APID 1366

NOAA Mission Specific Record, Appended Table 13

The final record worth noting is the Image Data Function Record. This record contains text data describing the image. This record is crucial for understanding the type of data contained in the packets. I’ve included a snapshot of the text contained in this record for the current APID.

APID 1366 Function Record

Image Data Function Record Text

The record informs us that the data represents calibrated infrared data (i.e. a temperature map), and maps the pixel values to the corresponding temperature. This is obviously invaluable data when interpreting the GOES images.

Following the header transmission is the data transmission. The data is transmitted row by row in a compressed format. The receiver must collect this data and isolate each row for decompression. For the 1366 APID, each row had, on average, 454 bytes. The final step in the process is to decompress these rows to retrieve the data.

When I originally performed this decoding I anticipated needing to implement the decompression myself. The data is compressed with Rice encoding and I kept expecting to find a packet which detailed the compression parameters, but I was never able to find one. It turns out, NOAA does not release the details of the compression algorithm. Luckily, there is a publicly available C++ library which contains routines to do the decompression. As a side note, there is an interesting paper describing deep space probes and the coding they’ve used. The paper contains is a nice discussion about how using Rice compression on the Voyager probe dramatically changed the acceptable bit error rate.

The required header, c++, and static libraries can be found here. I wrapped the decompression files into a mex file which I could call from the Matlab environment. I’ve included the Visual Studio Project I used to create this mex file. If you’ve never created a mex file before, it is not difficult at all. You may need to recompile the project to get it to run on your machine, but at a minimum the source code should provide an outline of how to proceed. Using this mex file, I obtained the following image for the APID 1366 data.

GOES Calibrated Infrared Image

┬áSo those are pretty much all the steps needed to produce an image from this GOES data. I’ve included all the required m-files as well as some data files containing the packets in various stages of processing. The data sets posted at SetiQuest contain quite a lot of interesting data. Not only do the LRIT files contain imaging data, but there is also weather reports and other text based information. The one thing the data sets didn’t contain was the data for a full disk image of the Earth, which is unfortunate since that would be very beautiful.

All the data and files can be downloaded here.