.net video voog MJPG moodi

Proovisin MJPG taolise protokolli teha c# keeles. Taolisega mõtlen ma, et see ei vasta mingitele standarditele. Aga põhimõtteliselt on olemas minimaalne klient ja server JPG piltide streamimiseks. Stream ise ei tee midagi frameide vahel, st. isegi mitte bitblt-i, nii et võib öelda, et intraframe tüüpi stream.

Ma tegin selle remote desktopi ühenduse moodi, aga ainult ühesuunalise. Põhimõtteliselt streamib klient desktopi pilte, mis on saadud sellise koodiga.

public class ScreenCapture
    {
        /// <summary>
        /// Creates an Image object containing a screen shot of the entire desktop
        /// </summary>
        /// <returns></returns>
        public Image CaptureScreen()
        {
            return CaptureWindow(User32.GetDesktopWindow());
        }
        /// <summary>
        /// Creates an Image object containing a screen shot of a specific window
        /// </summary>
        /// <param name="handle">The handle to the window. (In windows forms, this is obtained by the Handle property)</param>
        /// <returns></returns>
        public Image CaptureWindow(IntPtr handle)
        {
            // get te hDC of the target window
            IntPtr hdcSrc = User32.GetWindowDC(handle);
            // get the size
            User32.RECT windowRect = new User32.RECT();
            User32.GetWindowRect(handle, ref windowRect);
            int width = windowRect.right - windowRect.left;
            int height = windowRect.bottom - windowRect.top;
            // create a device context we can copy to
            IntPtr hdcDest = GDI32.CreateCompatibleDC(hdcSrc);
            // create a bitmap we can copy it to,
            // using GetDeviceCaps to get the width/height
            IntPtr hBitmap = GDI32.CreateCompatibleBitmap(hdcSrc, width, height);
            // select the bitmap object
            IntPtr hOld = GDI32.SelectObject(hdcDest, hBitmap);
            // bitblt over
            GDI32.BitBlt(hdcDest, 0, 0, width, height, hdcSrc, 0, 0, GDI32.SRCCOPY);
            // restore selection
            GDI32.SelectObject(hdcDest, hOld);
            // clean up 
            GDI32.DeleteDC(hdcDest);
            User32.ReleaseDC(handle, hdcSrc);
            // get a .NET image object for it
            Image img = Image.FromHbitmap(hBitmap);
            // free up the Bitmap object
            GDI32.DeleteObject(hBitmap);
            return img;
        }
        /// <summary>
        /// Captures a screen shot of a specific window, and saves it to a file
        /// </summary>
        /// <param name="handle"></param>
        /// <param name="filename"></param>
        /// <param name="format"></param>
        public void CaptureWindowToFile(IntPtr handle, string filename, ImageFormat format)
        {
            Image img = CaptureWindow(handle);
            img.Save(filename, format);
        }
        /// <summary>
        /// Captures a screen shot of the entire desktop, and saves it to a file
        /// </summary>
        /// <param name="filename"></param>
        /// <param name="format"></param>
        public void CaptureScreenToFile(string filename, ImageFormat format)
        {
            Image img = CaptureScreen();
            img.Save(filename, format);
        }

        /// <summary>
        /// Helper class containing Gdi32 API functions
        /// </summary>
        private class GDI32
        {

            public const int SRCCOPY = 0x00CC0020; // BitBlt dwRop parameter
            [DllImport("gdi32.dll")]
            public static extern bool BitBlt(IntPtr hObject, int nXDest, int nYDest,
                int nWidth, int nHeight, IntPtr hObjectSource,
                int nXSrc, int nYSrc, int dwRop);
            [DllImport("gdi32.dll")]
            public static extern IntPtr CreateCompatibleBitmap(IntPtr hDC, int nWidth,
                int nHeight);
            [DllImport("gdi32.dll")]
            public static extern IntPtr CreateCompatibleDC(IntPtr hDC);
            [DllImport("gdi32.dll")]
            public static extern bool DeleteDC(IntPtr hDC);
            [DllImport("gdi32.dll")]
            public static extern bool DeleteObject(IntPtr hObject);
            [DllImport("gdi32.dll")]
            public static extern IntPtr SelectObject(IntPtr hDC, IntPtr hObject);
        }

        /// <summary>
        /// Helper class containing User32 API functions
        /// </summary>
        private class User32
        {
            [StructLayout(LayoutKind.Sequential)]
            public struct RECT
            {
                public int left;
                public int top;
                public int right;
                public int bottom;
            }
            [DllImport("user32.dll")]
            public static extern IntPtr GetDesktopWindow();
            [DllImport("user32.dll")]
            public static extern IntPtr GetWindowDC(IntPtr hWnd);
            [DllImport("user32.dll")]
            public static extern IntPtr ReleaseDC(IntPtr hWnd, IntPtr hDC);
            [DllImport("user32.dll")]
            public static extern IntPtr GetWindowRect(IntPtr hWnd, ref RECT rect);
        }
    }

See on internetist leitud koodijupp, public domain igaljuhul. Niipalju kontrollisin seda koodi, et ta laseb kõik ressursid korralikult lahti. DeleteObject(), DeleteDC(). Kõige olulisemad jupid. Igaljuhul annab see klass ilusa bitmapi. Ainuke probleem on, et väiksemaks või suuremaks ei lase .net oma bitmappe teha minuarust. Selleks tuleb imelikku GDI koodi kirjutada, mida pole alati võimalik teha. Ei hakka ise ka interpoleerima seal seda suurust, võtsin Aforge.net Imaging.Filters variandi, sest mul oli see olemas.

Siis põhimõtteliselt on sellised protokolli klassid.

public class Dispatcher
    {
        public ClientProtocol proto;
        public bool IsRunning = false;
        public void Dispatch(TcpClient client, IPAddress address, int port , Logger logger)
        {
            try
            {
                client.Connect(address, port);
                Socket s = client.Client;
                proto = new ClientProtocol(s, logger);
                Thread t = new Thread(new ThreadStart(proto.HandleConnection));
                t.Start();
                IsRunning = true;
            }
            catch (Exception ex)
            {
                logger.WriteEntry(ex.ToString());
                IsRunning = false;
            }
        }
    }

See klass on tegelikult ainult Threadide tekitamiseks. Tavaliselt poleks Dispatcheril ClientProtokolliga mingit seost. Aga see on Singelton Dispatcher nö. Paremat nime ei oska leida.

Peamine osa koodist, mis “protokolliga” tegeleb on järgmises klassis.


public class ClientProtocol
    {
        public ClientProtocol(Socket s, Logger logger)
        {
            sender = s;
            this.logger = logger;
        }
        Socket sender;
        Logger logger;
        Queue<Bitmap> queue = new Queue<Bitmap>();
        Mutex mutex = new Mutex();
        int client_id;
        Random rand = new Random();
        BinaryFormatter binary_formatter = new BinaryFormatter();
        public bool running = false;
        public int debug_blobsize = 0;
        public void AddToQue(Bitmap b)
        {
            mutex.WaitOne();
            if (queue.Count > 100)
                queue.Clear();
            queue.Enqueue(b);
            mutex.ReleaseMutex();
        }
        Bitmap Dequeue()
        {
            mutex.WaitOne();
            Bitmap b = queue.Dequeue();
            mutex.ReleaseMutex();
            return b;
        }
        ImageCodecInfo GetEncoder(ImageFormat format)
        {
            ImageCodecInfo[] codecs = ImageCodecInfo.GetImageDecoders();

            foreach (ImageCodecInfo codec in codecs)
            {
                if (codec.FormatID == format.Guid)
                {
                    return codec;
                }
            }
            return null;

        }
        public void HandleConnection()
        {
            NetworkStream netstream = new NetworkStream(sender, false);
            ImageCodecInfo jpgEncoder = GetEncoder(ImageFormat.Jpeg);
            Encoder encoder = Encoder.Quality;
            EncoderParameters enc_params = new EncoderParameters(1);
            EncoderParameter enc_param = new EncoderParameter(encoder, 40L);//50 quality level compression(lower values = lower quality)
            enc_params.Param[0] = enc_param;
            running = true;
            while (running)
            {
                if (queue.Count > 0)
                {
                    try
                    {
                        Bitmap b = Dequeue();
                        
                        Dictionary<string, object> dict = new Dictionary<string, object>();
                        MemoryStream mems = new MemoryStream();
                        MemoryStream jpg = new MemoryStream();
                        b.Save(jpg, jpgEncoder, enc_params);
                        byte[] jpgarr = jpg.GetBuffer();
                        dict.Add("time", DateTime.Now.Second);
                        dict.Add("jpg", jpgarr);
                        binary_formatter.Serialize(mems, dict);
                        byte[] data = mems.GetBuffer();
                        int data_size = (int)mems.Length;
                        byte[] blobsize = BitConverter.GetBytes(data_size);
                        b.Dispose();
                        mems.Close();
                        jpg.Close();
                        if (netstream.CanWrite)
                        {
                            netstream.Write(blobsize, 0, 4);//blocking until all is written
                            netstream.Write(data, 0, data_size);
                            this.debug_blobsize = data_size;
                        }
                        else
                            throw new Exception();
                    }
                    catch (Exception ex)
                    {
                        //Console.Write(ex.ToString());
                        running = false;
                        return;
                    }
                }
            }
            netstream.Close();
            sender.Close();
            sender.Dispose();
        }

    }

Enamus huvitavat toimub HandleConnection meetodis. Kõigepealt tekitatakse JPEG encoder, mis on tuunitav. Kvaliteeti saab muuta täpselt niipalju kui ise tahad, mis on suhteliselt oluline bandwidth probleemide tekkimisel. Mul on hardcoded sisse see kvaliteedi väärtus. Ja üllatavalt hea kvaliteediga pildid tulevad läbi. Kuid päris videote vaatamiseks ei kõlba see klient, mis remote desk situatsioonis pole võibolla kõige olulisem. See ei ole muidugi selle Encoderi probleem enam, pigem lihtsalt veits rohkem vaeva vaja näha. Aga kõik ei ole nii halb.

Protokoll on lihtne. Kõigepealt loetakse edastatavate andmete suurus, see tuleb BitConverter.GetBytes(int x) meetodiga byteideks teha. Andmeteks on lihtsalt Dictionary objekt, mis on binary formatteriga serialiseeritud. Et JPEG on juba byte-ideks tehtud, siis binary formatter tegeleb ainult selle dictionary-ga, ja ei tee palju ülearust tööd. Pluss pool on see, et kui protokolli laiendada tahad, siis kirjutad sinna dictionary vastava laienduse juurde, ja serveris loed lihtsalt seda uut dictionary key value paari. Ma üritasin timestampi külge panna sinna dictionary, mida tegelikult server ei loe, aga sellest pole midagi, kõik töötab ikka. NetworkStream saadab kõigepealt dictionary pikkuse, mis antud juhul on JPEG pildi suurus enamjaolt, ja siis dictionary enda.

Selles klassis on veel Queue, mis töötab buffrina. Ma ei oska seda seletada, see on ball of mud minujaoks. Kuna threade kasutatakse, siis UI thread söödab neid pilte sinna Quesse. Mida see teine võrguga tegelev thread siis sööb. Pole võibolla parim lahendus, aga mina ei tea.

Vastuvõtja kood on siin klassis.

public class ClientProtocol
    {
        public ClientProtocol(Socket s)
        {
            reciever = s;
        }
        public Queue<Bitmap> queue = new Queue<Bitmap>();
        Mutex mutex = new Mutex();
        Socket reciever;
        public bool running = true;
        BinaryFormatter binaryf = new BinaryFormatter();
        void Enque(Bitmap b)
        {
            mutex.WaitOne();
            if (queue.Count > 100)
            {
                queue.Clear();
            }
            queue.Enqueue(b);
            mutex.ReleaseMutex();
        }
        public Bitmap Dequeue()
        {
            mutex.WaitOne();
            Bitmap b = queue.Dequeue();
            mutex.ReleaseMutex();
            return b;
        }
        public void HandleConnection()
        {
            NetworkStream netstream = new NetworkStream(reciever);
            while (running)
            {
                try
                {
                    MemoryStream memstream = new MemoryStream();
                    byte[] blobsize = new byte[4];
                    int recv = netstream.Read(blobsize, 0, 4);
                    if (recv != 4)
                        throw new Exception("no size");
                    int blobsz = BitConverter.ToInt32(blobsize, 0);
                    int offset = 0;

                    while (blobsz > 0)
                    {
                        byte[] data = new byte[blobsz];
                        recv = netstream.Read(data, 0, blobsz);
                        blobsz -= recv;
                        memstream.Write(data, 0, recv);
                    }
                    memstream.Position = 0;
                    Dictionary<string, object> dict = (Dictionary<string, object>)binaryf.Deserialize(memstream);
                    memstream.Close();
                    byte[] jpgbytes = (byte[])dict["jpg"];
                    MemoryStream memjpg = new MemoryStream(jpgbytes);

                    Enque((Bitmap)Image.FromStream(memjpg));
                    memjpg.Close();
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.Message);
                    Console.WriteLine();
                }
            }
        }
    }

Õige viis kuidas NetworkStreami lugeda on see koodijupp

while (blobsz > 0)
                    {
                        byte[] data = new byte[blobsz];
                        recv = netstream.Read(data, 0, blobsz);
                        blobsz -= recv;
                        memstream.Write(data, 0, recv);
                    }

sest recv võib suvaline suurus olla, tuleb midagi sarnast teha, mis esialgu jäi arusaamatuks mulle vähemalt.

Selles klassis on jälle üks Queue, millest UI loeb pilte ja näitab. Ma kasutasin tavalist pildi kasti, mis kindlasti pole video jaoks mõeldud, aga see töötas suht videona. Ma ei tea kas mingi spets kast annaks juurde midagi.

Ja mudball dispatcher klass.

public class Dispatcher
    {
        public ClientProtocol proto;
        public void Dispatch(TcpListener client)
        {
            try
            {
                client.Start();
                Socket s = client.AcceptSocket();//block waiting for connection
                proto = new ClientProtocol(s);
                Thread t = new Thread(proto.HandleConnection);
                t.Start();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }
    }

Saatja UI koodis pole midagi muud kui kell tiksuma panna ja dispatcher.proto.addtoqueue();

Vastuvõtjas dispatcher.proto.dequeue();

Üks asi, mis “päris” videot vaadates silma hakkab on, et kui suvalisel kohal pausile panna, siis frame-ide vahel on nö. vahepealsed frame-id. Need vahepealsed pildid koosnevad eelmise ja järgmise pildi liitmisest. Ma ei tea kuidas seda nimetatakse, aga nii saaks FPS-i paremaks suht odava trikiga.

Advertisements