Network Video Analytics in C#: How to achieve line detection (NVA – Object Detection)

February 10, 2015 LeonDunn11 .Net 4, C# - Image Processing, C# Advanced 0

The importance of surveillance and security camera systems is constantly growing these days. In the ancient times, there was a camera and a guard whose main task was to stare at the monitor and try not to fall asleep. As technology is developing day-by-day, new technologies arise to help the people who are in need of a reliable security system. One of these new technologies is object detection.
Inside object detection, there are the so-called feature-based methods. These methods aim to find possible matches between the features of the object and the image of the camera. The topic of this tutorial belongs to these feature-based methods.
The following tutorial aims to introduce line detection. First, you can read a brief theoretical background about object detection in general and line detection. After this part you will find a detailed description on how to implement this exciting function in C# along with creating a graphical user interface for the program.
I hope this article will help you in developing your own application to perform line detection.

Line Detection: What will you need to succeed?

Before we start programming, let’s see what devices are needed for this game.

Obviously, it is necessary to have a USB camera device to work with. You can consult the webpage for USB cameras; there are some pretty nice ones there.

Regarding the software part of the business, you need to download Visual C# 2010 Express and .Net Framework 4. Those are needed for the coding procedure.

One final part is missing now: a camera SDK that you can use for developing your line detecting solution. I found this software, Ozeki’s Camera SDK to be useful, it was easy to work with, and so I can recommend it to you.

Now that you are fully prepared to start working, take a look at the literature.

What is object detection and line detection?

To get a deeper knowledge about object detection, we can consult our old friend, the Internet. As it is stated on Wikipedia, “object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. Well-researched domains of object detection include face detection and pedestrian detection.”

Another source, Mathworks describes object detection as “the process of finding instances of real-world objects such as faces, bicycles, and buildings in images or videos.” As possible fields of usage, the webpage lists image retrieval, security, surveillance, and automated vehicle parking systems.

If we dig deeper in this topic we can find line detection. The basic task of line detection is to indicate if an object on the camera’s image crosses any of the virtual lines we have on our image. You can use this function for security purposes or, for example, to observe costumer traffic in your shop.

Typical Application Areas
Typical Application Areas

Fine, now here comes the exciting part of the story. Let’s see how to implement this function.

Creating the C# code

To detect lines on the image of a camera, we should use the ILinedetector object. After creating an instance of ILineDetector with the help of the static ImageProcesserFactory class, it will be able to detect lines on frames and on videos, too. In case of frames, we can make the image with the help of the Process() method of the instance, while in the case of videos, we should use the ImageProcesserHandler mediahandler.

After this tiny piece of information, let the actual work begin!

As the first step of coding we should apply the using lines.

using System;
using System.Drawing;
using System.Windows.Forms;
using Ozeki.Media.MediaHandlers;
using Ozeki.Media.MediaHandlers.Video;
using Ozeki.Media.MediaHandlers.Video.CV;
using Ozeki.Media.MediaHandlers.Video.CV.Data;
using Ozeki.Media.MediaHandlers.Video.CV.Processer;
using Ozeki.Media.Video.Controls;

Then we qualify the namespace in our code.

namespace LineDetection
{

After these basic steps, it is necessary to add the global variables to our code. Here, you will need Webcamera_webCamera. This instance will help us to get the image of the camera. The next variable is MediaConnector_connector. Its task is to connect the mediahandlers. Now, here comes ImageProcesserHandler_imageProcesserHandler. This guy is a mediahandler that runs the instances that implements the IImageProcesser interface. Now we have the interface that is responsible for line detection and that implements the IImageProcesser interface: ILineDetector_lineDetector. Three more variables to go. FrameCapture_frameCapture is another mediahandler. You can configure the frequency of processing with this one. We have the VideoViewerWF instance that is a GUI tool for Windows Forms applications and it will help you to display the video. And last but not least, we have the DrawingImageProvider instance that is also a mediahandler. (Mediahandlers… mediahandlers everywhere.) Its task is to prepare the image, sent by the VideoSender class mediahandlers, for the VideoViewerWF instance. Now we have all the necessary global variables set in our code. Let’s see how it looks like now.

public partial class Form1 : Form{
WebCamera _webCamera;
MediaConnector _connector;
ImageProcesserHandler _imageProcesserHandler;
ILineDetector _lineDetector;
FrameCapture _frameCapture;
VideoViewerWF _originalView;
VideoViewerWF _processedView;
DrawingImageProvider _originalImageProvider;
DrawingImageProvider _processedImageProvider;

We are done with the global variables for now, so we can move on to some methods that should be called to work out line detecting.

The first of these methods is Init(). This is the method that initializes most of the global variables. This is where the FrameCapture mediahandler instance is configured and also, where the ILineDetector instance is created with the help of ImageProcesserFactory. We add this instance to the ImageProcesserHandler instance. Every time the ILineDetector instance processes an image, it will be indicated by the DetectionOccured event which we can subscribe for here, too.


void Init()
{
_frameCapture = new FrameCapture();
_frameCapture.SetInterval(5);
_webCamera = WebCamera.GetDefaultDevice();
_connector = new MediaConnector();
_originalImageProvider = new DrawingImageProvider();
_processedImageProvider = new DrawingImageProvider();
_lineDetector = ImageProcesserFactory.CreateLineDetector();
_lineDetector.DetectionOccurred += _lineDetector_DetectionOccurred;
_imageProcesserHandler = new ImageProcesserHandler();
_imageProcesserHandler.AddProcesser(_lineDetector);
}

Our second method is the SetVideoViewers() method, that generates and initializes the objects that are responsible for displaying the video. It defines the VideoViewerWF instances, sets their properties, assigns the proper DrawingImageProvider instances and adds them to the GUI.


void SetVideoViewers()
{
_originalView = new VideoViewerWF
{
BackColor = Color.Black,
Location = new Point(10, 20),
Size = new Size(320, 240)
};

_originalView.SetImageProvider(_originalImageProvider);
Controls.Add(_originalView);

_processedView = new VideoViewerWF
{
BackColor = Color.Black,
Location = new Point(350, 20),
Size = new Size(320, 240)
};

_processedView.SetImageProvider(_processedImageProvider);
Controls.Add(_processedView);
}

We should not forget about the InvokeGUIThread() method. This is the method that handles the GUI thread. It executes the specified method asynchronously on the thread that the control’s underlying handle was created on.


void InvokeGUIThread(Action action)
{
BeginInvoke(action);
}

The next method, the InitDetectorFields() fills in the Text Boxes on the GUI with the configurations of the ILineDetector with the help of the InvokeGUIThread() helper method.

void InitDetectorFields()
{
InvokeGuiThread(() =>
{
chk_ShowImage.Checked = _lineDetector.ShowImage;
tb_Red.Text = _lineDetector.DrawColor.R.ToString();
tb_Green.Text = _lineDetector.DrawColor.G.ToString();
tb_Blue.Text = _lineDetector.DrawColor.B.ToString();
tb_DrawThickness.Text = _lineDetector.DrawThickness.ToString();

tb_AngleResolution.Text = _lineDetector.AngleResolution.ToString();
tb_CannyThreshold.Text = _lineDetector.CannyThreshold.ToString();
tb_CannyThresholdLinking.Text = _lineDetector.CannyThresholdLinking.ToString();
tb_DistanceResolution.Text = _lineDetector.DistanceResolution.ToString();
tb_LineGap.Text = _lineDetector.LineGap.ToString();
tb_LineWidth.Text = _lineDetector.LineWidth.ToString();
tb_Threshold.Text = _lineDetector.Threshold.ToString();
});
}

Moving on, we have the ConnectWebcam() method. This one is responsible for connecting the proper mediahandler instances with the help of the MediaConnector instance. One ImageProvider object receives the original image of the camera, while the other one receives the processed image.


void ConnectWebcam()
{
_connector.Connect(_webCamera, _originalImageProvider);

_connector.Connect(_webCamera, _frameCapture);
_connector.Connect(_frameCapture, _imageProcesserHandler);
_connector.Connect(_imageProcesserHandler, _processedImageProvider);

After getting through all the initialization, the mediahandlers can start operating. Now, the Start() method will help us.


void Start()
{
_originalView.Start();
_processedView.Start();

_frameCapture.Start();
_webCamera.Start();
}

If you press the Set button that belongs to the Group Box on the GUI (we’ll talk about it a little later), the following event will be called whose task is to configure the LineDetector.


void btn_Set_Click(object sender, EventArgs e)
{
InvokeGUIThread(() =>
{
_lineDetector.AngleResolution = Double.Parse(tb_AngleResolution.Text);
_lineDetector.CannyThreshold = Double.Parse(tb_CannyThreshold.Text);
_lineDetector.CannyThresholdLinking = Double.Parse(tb_CannyThresholdLinking.Text);
_lineDetector.DistanceResolution = Double.Parse(tb_DistanceResolution.Text);
_lineDetector.LineGap = Double.Parse(tb_LineGap.Text);
_lineDetector.LineWidth = Double.Parse(tb_LineWidth.Text);
_lineDetector.Threshold = Int32.Parse(tb_Threshold.Text);
});
}

Let’s take a closer look to see what those values in this snippet are responsible for.
At AngleResolution you can set the resolution in the angle area.
CannyThreshold determines the value of thresholding to find initial segments of strong edges.
The CannyThresholdLinking value is used to determine the number of pixels in the edges of an image.
DistanceResolution defines the resolution between pixel-related units.
LineGap indicates the minimum gap between lines detectable lines.
LineWidth indicates the minimum width of detectable lines.
For a line to be actually considered as detected, it is necessary to reach the value of Threshold.

Post Process settings

After the detection happened, you can configure how the output image should behave, what changes it shall undergo. You can configure these features with the help of the following settings.
With the help of the ShowImage checkbox you can set if you only want to see the processed image with the detected shapes in the programme or you also want to have the original image.
You can set the colour of the marking of the detected objects with DrawColor.
With DrawThickness you can set the thickness of the marking of the detected objects.
These settings are executed by the Set button on the highlight group box. If you click on this button, the following event will be called:


void btn_HighlightSet_Click(object sender, EventArgs e)
{
InvokeGUIThread(() =>
{
_lineDetector.ShowImage = chk_ShowImage.Checked;
_lineDetector.DrawColor = Color.FromArgb(Int32.Parse(tb_Red.Text), Int32.Parse(tb_Green.Text), Int32.Parse(tb_Blue.Text));
_lineDetector.DrawThickness = Int32.Parse(tb_DrawThickness.Text);
});
}

Detection

After processing each image/frame, the DetectionOccured event will pop up.

void _lineDetector_DetectionOccurred(object sender, LineDetectedEventArgs e)
{
InvokeGUIThread(() =>
{
lb_Detection.Items.Clear();

foreach (var info in e.Info)
{
lb_Detection.Items.Add(info);
}
});
}

You can find the list of the detected lines in the arguments of this event and in that list you can query the starting and endpoint, the direction and the length of each line.

Creating the GUI

Of course, a nice user interface will be needed in order to manage the camera and line detection properly. I will provide you with some code snippets I used to create my GUI. As it can be seen on the picture, my GUI is built from 4 parts. The first part is where the original and the processed pictures are shown. Under this you can see the field where the list of detections is created. Two group boxes can be seen on the right, one for highlights and one for settings.

line detection gui
Now take a closer look at the different parts of the user interface.
Let’s start with the two images. On the left side you can see the original image of the camera.


this.label1.AutoSize = true;
this.label1.Font = new System.Drawing.Font("Microsoft Sans Serif", 8.25F, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, ((byte)(238)));
this.label1.Location = new System.Drawing.Point(30, 265);
this.label1.Name = "label1";
this.label1.Size = new System.Drawing.Size(87, 13);
this.label1.TabIndex = 0;
this.label1.Text = "Original image";

On the right side there is the processed image.

this.label2.AutoSize = true;
this.label2.Font = new System.Drawing.Font("Microsoft Sans Serif", 8.25F, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, ((byte)(238)));
this.label2.Location = new System.Drawing.Point(370, 265);
this.label2.Name = "label2";
this.label2.Size = new System.Drawing.Size(103, 13);
this.label2.TabIndex = 1;
this.label2.Text = "Processed image";

The Set button under Settings was mentioned before with which you can configure the basic settings in your program. First, let’s see how to create that almighty Set button.

this.btn_Set.Location = new System.Drawing.Point(189, 368);
this.btn_Set.Name = "btn_Set";
this.btn_Set.Size = new System.Drawing.Size(58, 23);
this.btn_Set.TabIndex = 2;
this.btn_Set.Text = "Set";
this.btn_Set.UseVisualStyleBackColor = true;
this.btn_Set.Click += new System.EventHandler(this.btn_Set_Click);

Now we have the button. But if we want to adjust any settings with it, we need some parameters that we can adjust

Let’s start with Angle Resolution.

this.tb_AngleResolution.Location = new System.Drawing.Point(138, 26);
this.tb_AngleResolution.Name = "tb_AngleResolution";
this.tb_AngleResolution.Size = new System.Drawing.Size(87, 20);
this.tb_AngleResolution.TabIndex = 3;

Now the code for the Canny Threshold comes.

this.tb_CannyThreshold.Location = new System.Drawing.Point(138, 58);
this.tb_CannyThreshold.Name = "tb_CannyThreshold";
this.tb_CannyThreshold.Size = new System.Drawing.Size(87, 20);
this.tb_CannyThreshold.TabIndex = 4;

And do not forget about Canny Threshold Linking.

this.tb_CannyThresholdLinking.Location = new System.Drawing.Point(138, 90);
this.tb_CannyThresholdLinking.Name = "tb_CannyThresholdLinking";
this.tb_CannyThresholdLinking.Size = new System.Drawing.Size(87, 20);
this.tb_CannyThresholdLinking.TabIndex = 5;

Or about Distance Resolution.

this.tb_DistanceResolution.Location = new System.Drawing.Point(138, 122);
this.tb_DistanceResolution.Name = "tb_DistanceResolution";
this.tb_DistanceResolution.Size = new System.Drawing.Size(87, 20);
this.tb_DistanceResolution.TabIndex = 6;

Almost done. Here comes the Line Gap.

this.tb_LineGap.Location = new System.Drawing.Point(138, 154);
this.tb_LineGap.Name = "tb_LineGap";
this.tb_LineGap.Size = new System.Drawing.Size(87, 20);
this.tb_LineGap.TabIndex = 7;

And the Line Widht.

this.tb_LineWidth.Location = new System.Drawing.Point(138, 186);
this.tb_LineWidth.Name = "tb_LineWidth";
this.tb_LineWidth.Size = new System.Drawing.Size(87, 20);
this.tb_LineWidth.TabIndex = 8;

And finally, the code for Threshold.

this.tb_Threshold.Location = new System.Drawing.Point(138, 218);
this.tb_Threshold.Name = "tb_Threshold";
this.tb_Threshold.Size = new System.Drawing.Size(87, 20);
this.tb_Threshold.TabIndex = 9;

These are the different settings you can adjust in your program. After we are ready creating the boxes where you can provide the values for these parameters, we can go on.

As it was mentioned above, you will have a list of the detected lines. Here is how you can create the field for the list.

this.lb_Detection.FormattingEnabled = true;
this.lb_Detection.Location = new System.Drawing.Point(10, 322);
this.lb_Detection.Name = "lb_Detection";
this.lb_Detection.Size = new System.Drawing.Size(660, 251);
this.lb_Detection.TabIndex = 14;

Let’s go on with the other group box with of Highlights.

This part also has got its Set button (in the code it has the name of HighlightSet to differentiate it from the other Set button).

this.btn_HighlightSet.Location = new System.Drawing.Point(189, 129);
this.btn_HighlightSet.Name = "btn_HighlightSet";
this.btn_HighlightSet.Size = new System.Drawing.Size(58, 23);
this.btn_HighlightSet.TabIndex = 19;
this.btn_HighlightSet.Text = "Set";
this.btn_HighlightSet.UseVisualStyleBackColor = true;
this.btn_HighlightSet.Click += new System.EventHandler(this.btn_HighlightSet_Click);

And two parameters to provide.

this.tb_DrawThickness.Location = new System.Drawing.Point(138, 95);
this.tb_DrawThickness.Name = "tb_DrawThickness";
this.tb_DrawThickness.Size = new System.Drawing.Size(87, 20);
this.tb_DrawThickness.TabIndex = 17;

.tb_Blue.Location = new System.Drawing.Point(183, 59);
this.tb_Blue.Name = “tb_Blue”;
this.tb_Blue.Size = new System.Drawing.Size(42, 20);
this.tb_Blue.TabIndex = 16;

this.tb_Green.Location = new System.Drawing.Point(138, 59);
this.tb_Green.Name = “tb_Green”;
this.tb_Green.Size = new System.Drawing.Size(42, 20);
this.tb_Green.TabIndex = 15;

this.tb_Red.Location = new System.Drawing.Point(93, 59);
this.tb_Red.Name = “tb_Red”;
this.tb_Red.Size = new System.Drawing.Size(42, 20);
this.tb_Red.TabIndex = 14;

There is one checkbox here to set whether we want to see the original image or not:


this.chk_ShowImage.AutoSize = true;
this.chk_ShowImage.CheckAlign = System.Drawing.ContentAlignment.MiddleRight;
this.chk_ShowImage.Location = new System.Drawing.Point(22, 25);
this.chk_ShowImage.Name = "chk_ShowImage";
this.chk_ShowImage.Size = new System.Drawing.Size(85, 17);
this.chk_ShowImage.TabIndex = 14;
this.chk_ShowImage.Text = "ShowImage:";
this.chk_ShowImage.UseVisualStyleBackColor = true;

At the end here you can see the Main form of the GUI.


this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
this.ClientSize = new System.Drawing.Size(947, 581);
this.Controls.Add(this.label12);
this.Controls.Add(this.lb_Detection);
this.Controls.Add(this.groupBox2);
this.Controls.Add(this.groupBox1);
this.Controls.Add(this.label2);
this.Controls.Add(this.label1);
this.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle;
this.MaximizeBox = false;
this.Name = "MainForm";
this.Text = "Line Detection";
this.Load += new System.EventHandler(this.MainForm_Load);
this.groupBox1.ResumeLayout(false);
this.groupBox1.PerformLayout();
this.groupBox2.ResumeLayout(false);
this.groupBox2.PerformLayout();
this.ResumeLayout(false);
this.PerformLayout();

And we are done! It was fun, wasn’t it?

Conclusion

Now line detecting is ready to be used! In this tutorial you could read some theoretic background about implementing line detecting in C# and a detailed practical description about this solution, including some code examples. The article also contains a detailed description about implementing a useful user interface for a program like this.
I hope this brief tutorial was useful for you all. Good luck on developing your own solutions!

References

Attachments

Please click here to download the Example Project for Line detection in Csharp