Home ForumsSoftwareSDK Frameworks Thread
Confused about converting Raw Depth values to a distance in C#
 
btshrewsbury
Posted: 14 February 2011 02:15 PM   [ Ignore ]
New Member
Rank
Total Posts:  6
Joined  2011-02-14

Howdy,

I’m having trouble pulling the depth values and converting them to a usable distance. I’m pretty bad at image manipulations….


I pulled the image data out of memory and converted it to an array of bytes. I’m getting what looks like argb. Then i’m trying to convert that to greyscale using greyscaleArray[s] = 0.2989 * imageAsBytes[x+1] + 0.5870 * imageAsBytes[x+2] + 0.1140 * imageAsBytes[x+3];

once I get that I move on to resizing the array to a smaller size. 640*480 is way to big for our application. I’m not sure if my averaging is off or the greyscale. I would really appreciate some info on how to do this correctly. Preferably in C# but beggars really can’t be choosers!

Cheers

public static int[,] mapZValues(IntPtr pointerToImage)
        {
            
            byte[] imageAsBytes = new byte[(uint)640 * (uint)480 * 4];
            Marshal.Copy(pointerToImage, imageAsBytes, 0, 640 * 480 * 4);

            //Try to convert the image to greyscale
            double[] greyscaleArray = new double[imageAsBytes.Length / 4];
            int s = 0;
            for (int x = 0; x < imageAsBytes.Length; x = x+4)
            {
                    greyscaleArray[s] = 0.2989 * imageAsBytes[x+1] + 0.5870 * imageAsBytes[x+2] + 0.1140 * imageAsBytes[x+3];
                    s++;
            }
           
            //Create a 2d array to store the greyscale converted pixels
            double[,] depth2dArray = new double[640, 480];

            for (int x = 0; x < 640; x++)
            {
                for (int y = 0; y < 480; y++)
                {
                    depth2dArray[x, y] = Math.Tan(greyscaleArray[y + (x * 480)] / 1024 + 0.5) * 33.825 + 5.7;
                }
            }
            
            return averageMatrix(depth2dArray,2,2);
        }


        //I need a resized version of the array (smaller version)
        public static int[,] averageMatrix(double[,] matrix, int newWidth, int newHeight)
        {
            double[] singleDimensionTempArray = new double[newWidth * newHeight];
            int[,] newDepthArray = new int[newWidth, newHeight];
            double average = 0;

            if (newWidth * newHeight < matrix.Length && matrix != null)
            {
                int widthOfSubGridToAverage = matrix.GetLength(0) / newWidth;
                int heightOfSubGridToAverage = matrix.Length / matrix.GetLength(0) / newHeight;

                for (int i = 0; i < matrix.Length / (widthOfSubGridToAverage * heightOfSubGridToAverage); i++)
                {
                    for (int j = 0; j < widthOfSubGridToAverage; j++)
                    {
                        int currentColumn = (j + (i * widthOfSubGridToAverage) % matrix.GetLength(0));
                        for (int k = 0; k < heightOfSubGridToAverage; k++)
                        { 
                            int currentRow = (k + (i * heightOfSubGridToAverage) % (matrix.Length / matrix.GetLength(0)));
                            average += matrix[currentColumn, currentRow];
                        }
                    }
                    singleDimensionTempArray[i] = average / (widthOfSubGridToAverage * heightOfSubGridToAverage);
                    average = 0;
                }
            }

            for (int x = 0; x < newWidth; x++)
            {
                for (int y = 0; y < newHeight; y++)
                {
                    newDepthArray[x, y] = (int) singleDimensionTempArray[y + (x * newWidth)];
                }
            }

            return newDepthArray;
        } 
Profile
 
 
thesmileman
Posted: 17 February 2011 07:52 PM   [ Ignore ]   [ # 1 ]
Jr. Member
RankRank
Total Posts:  47
Joined  2010-02-15

Since as you say you are not good at image manipulations why not use OpenCV (or any other image library for that matter)? There should be bindings for C# for OpenCV.

I have some example code here in the forums that shows how to get the RGB data and the depth both as a greyscale image and color image. Also resizing in OpenCV is only one line of code (As it is in most libraries).

Here is the post which has full source for the code (don’t worry it is only about 20 lines of actual code for the whole thing including the display)
http://codelaboratories.com/forums/viewthread/518/

On another note: Unfortunately I don’t have time to fully look at what you are doing but you seem to be doing a lot of work to just get the depth pixels and then scale them down. You would get much better results with a decent scaling algorithm (OpenCV hint hint )

Profile
 
 
btshrewsbury
Posted: 23 February 2011 01:39 PM   [ Ignore ]   [ # 2 ]
New Member
Rank
Total Posts:  6
Joined  2011-02-14
thesmileman - 17 February 2011 07:52 PM

Since as you say you are not good at image manipulations why not use OpenCV (or any other image library for that matter)? There should be bindings for C# for OpenCV.

I have some example code here in the forums that shows how to get the RGB data and the depth both as a greyscale image and color image. Also resizing in OpenCV is only one line of code (As it is in most libraries).

Here is the post which has full source for the code (don’t worry it is only about 20 lines of actual code for the whole thing including the display)
http://codelaboratories.com/forums/viewthread/518/

On another note: Unfortunately I don’t have time to fully look at what you are doing but you seem to be doing a lot of work to just get the depth pixels and then scale them down. You would get much better results with a decent scaling algorithm (OpenCV hint hint )

Thanks for the tip sir. I decided to try my luck with AForge since OpenCV isn’t a .net language.

I’m actually making some more headway working with the raw depth data

public static BitmapSource createGreyscaleDepthImage(IntPtr pointerToImage)
        {
            byte[] imageAsBytes = new byte[640 * 480 * 2];
            Marshal.Copy(pointerToImage, imageAsBytes, 0, 640 * 480 * 2);

            byte[] smallerByteArray = new byte[640 * 480];
            

            int smallerArrayIndex = 0;
            for (int i = 0; i < 640 * 480 * 2; i = i + 2)
            {
                smallerByteArray[smallerArrayIndex] = (byte) ((imageAsBytes[i] + (imageAsBytes[i] << 8)) / 8);
                smallerArrayIndex++;
            }
            BitmapSource bmp = BitmapSource.Create(640, 480, 96, 96, System.Windows.Media.PixelFormats.Gray8, null, smallerByteArray, 640);
            bmp.Freeze();

            return bmp;


        } 

It turns out I really only need relative distances and not exact measurements for my experiment. So I decided to try to manually convert the image to 8bit grayscale.
Capture2.jpg

Anybody have any idea why it looks so pinstripe - ish?

Profile
 
 
btshrewsbury
Posted: 23 February 2011 02:05 PM   [ Ignore ]   [ # 3 ]
New Member
Rank
Total Posts:  6
Joined  2011-02-14
public static BitmapSource createGreyscaleDepthImage(IntPtr pointerToImage)
        {
            byte[] imageAsBytes = new byte[640 * 480 * 2];
            Marshal.Copy(pointerToImage, imageAsBytes, 0, 640 * 480 * 2);

            byte[] smallerByteArray = new byte[640 * 480];
            

            int smallerArrayIndex = 0;
            for (int i = 0; i < 640 * 480 * 2; i = i + 2)
            {
                smallerByteArray[smallerArrayIndex] = (byte) ((imageAsBytes[i] + (imageAsBytes[i+1] << 8)) / 8);
                smallerArrayIndex++;
            }
            BitmapSource bmp = BitmapSource.Create(640, 480, 96, 96, System.Windows.Media.PixelFormats.Gray8, null, smallerByteArray, 640);
            bmp.Freeze();

            return bmp;


        } 

HA HAA! I had

imageAsBytes[i] + (imageAsBytes[i] << 8) instead of imageAsBytes[i] + (imageAsBytes[i+1] << 8) 


Capture3.JPG

Profile
 
 
thesmileman
Posted: 23 February 2011 09:17 PM   [ Ignore ]   [ # 4 ]
Jr. Member
RankRank
Total Posts:  47
Joined  2010-02-15

are you loosing precision in your conversion?

Profile
 
 
btshrewsbury
Posted: 24 February 2011 05:02 AM   [ Ignore ]   [ # 5 ]
New Member
Rank
Total Posts:  6
Joined  2011-02-14

Yea, I scaled the depth data down to a single byte. I only get 256 colors for depth. I don’t need great resolution for my project. But if anybody wants thhe full resolution just take out the divide by 8 and change the pixel type

Profile
 
 
btshrewsbury
Posted: 24 February 2011 06:50 AM   [ Ignore ]   [ # 6 ]
New Member
Rank
Total Posts:  6
Joined  2011-02-14

Here are the two functions cleaned up. Maybe they can help someone. They are in C# and are geared for WPF.
The first creates a 16bit grayscale image without any loss. It scales across the full 16bit rather crudely.
The second goes the opposite direction.  It scales the image down to 8bits grayscale by adding the depth data together and dividing it by 8. To make it faster you can shift the bits instead of dividing or multiplying. I figured Normal math was easier to understand.

/// <summary>
        /// Accepts the IntPtr object from thr NUIImage class and generates a 16bit grayscale image
        /// The image is poorly expanded to fill the 16 bits
        /// </summary>
        /// <param name="pointerToImage">The pointer to the RAW depth data</param>
        /// <returns>An image wpf can easily manipulate (basically the same object type we began with)</returns>
        public BitmapSource create16bitGreyscaleDepthImage(IntPtr pointerToImage)
        {
            //Requires you to use the RAW depth data type
            //Effects: Pulls the image out of memory and throws it inside an array we can use
            //Creates a new Unsigned short array with the raw depth data added togethor and scaled
            //Generates a new greyscale image from the unsigned short array
  
            byte[] imageAsBytes = new byte[640 * 480 * 2];
            Marshal.Copy(pointerToImage, imageAsBytes, 0, 640 * 480 * 2);

            ushort[] smallerShortArray = new ushort[640 * 480];


            //Pulls the depth data out (2 bytes) and adds the two values togethor. Multiplies by 32 to scale the image across all 16 bits
            int smallerArrayIndex = 0;
            for (int i = 0; i < 640 * 480 * 2; i = i + 2)
            {
                smallerShortArray[smallerArrayIndex] = (ushort) (imageAsBytes[i] + (imageAsBytes[i + 1] << 8) * 32);
                smallerArrayIndex++;
            }

            //Creates the new grayscale image
            BitmapSource bmp = BitmapSource.Create(640, 480, 96, 96, System.Windows.Media.PixelFormats.Gray16, null, smallerShortArray, 640*2);
            bmp.Freeze();

            return bmp;
        } 
/// <summary>
        /// Accepts the IntPtr object from thr NUIImage class and generates an 8bit grayscale image
        /// The resulting image looses resolution and is scaled down to 1 byte per pixel
        /// </summary>
        /// <param name="pointerToImage">The pointer to the RAW depth data</param>
        /// <returns>An image wpf can easily manipulate (basically the same object type we began with)</returns>
        public BitmapSource create8bitGreyscaleDepthImage(IntPtr pointerToImage)
        {
            //Requires you to use the RAW depth data type
            //Effects: Pulls the image out of memory and throws it inside an array we can use
            //Creates a new byte array with the raw depth data added togethor and scaled
            //Generates a new greyscale image from the byte array

            byte[] imageAsBytes = new byte[640 * 480 * 2];
            Marshal.Copy(pointerToImage, imageAsBytes, 0, 640 * 480 * 2);

            byte[] smallerByteArray = new byte[640 * 480];

            //Pulls the depth data out (2 bytes) and adds the two values togethor.Divides by 8 to scale the image down to 8 bits
            int smallerArrayIndex = 0;
            for (int i = 0; i < 640 * 480 * 2; i = i + 2)
            {
                smallerByteArray[smallerArrayIndex] = (byte) ((imageAsBytes[i] + (imageAsBytes[i+1] << 8)) / 8);
                smallerArrayIndex++;
            }

            //Creates the new grayscale image
            BitmapSource bmp = BitmapSource.Create(640, 480, 96, 96, System.Windows.Media.PixelFormats.Gray8, null, smallerByteArray, 640);
            bmp.Freeze();

            return bmp;
        } 


Take a look and try it out. Let me know if you see any errors or experience performance issues.

Profile
 
 
 
 
RSS 2.0     Atom Feed
Home ForumsSoftwareSDK Frameworks Thread