JeVoisBase  1.20
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
ColorFiltering.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2017 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
19 #include <jevois/Types/Enum.H>
20 #include <jevois/Debug/Timer.H>
22 
23 #include <opencv2/imgproc/imgproc.hpp>
24 
31 
32 // icon by Freepik in other at flaticon
33 
34 static jevois::ParameterCategory const ParamCateg("Color Filtering Parameters");
35 
36 //! Enum \relates ColorFiltering
37 JEVOIS_DEFINE_ENUM_CLASS(Effect, (NoEffect) (Blur) (Median) (Morpho) (Laplacian) (Bilateral) );
38 
39 //! Parameter \relates ColorFiltering
40 JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(effect, Effect, "Image processing effect to apply",
41  Effect::Morpho, Effect_Values, ParamCateg);
42 
43 //! Image filtering using OpenCV
44 /*! This module is to learn about basic image filtering. It was developed to allow students to instantly observe the
45  effects of different filters and their parameters on live video. The module implements a variety of filters using
46  OpenCV. Each filter exposes some paremeters (e.g., kernel size) that can be set interactively to understand their
47  effects onto the filter behavior.
48 
49  Available filters:
50 
51  - Blur: Replace each pixel by the average of all pixels in a box around the pixel of interest. Leads to a
52  blur effect on the image, more pronounced with bigger box sizes. See
53  http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#ga8c45db9afe636703801b0b2e440fce37
54 
55  - Median: Replaces each pixel by the median value within a box around that pixel. Tends to blur and remove
56  salt-and-peper noise from the
57  image. http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#ga564869aa33e58769b4469101aac458f9
58 
59  - Morpho: mathematical morphology operations, see
60  http://docs.opencv.org/3.2.0/d9/d61/tutorial_py_morphological_ops.html for an introduction,
61  and http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html
62 
63  - Laplacian: Computes second spatial derivative of the image. Tends to amplify edges and noise. See
64  http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#gad78703e4c8fe703d479c1860d76429e6
65 
66  - Bilateral: bi-lateral filter, very slow, see
67  http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/MANDUCHI1/Bilateral_Filtering.html and
68  http://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#ga9d7064d478c95d60003cf839430737ed
69 
70  How to use this module
71  ----------------------
72 
73  - On JeVois-Pro, select the effect and set its parameters in the Parameters tab of the GUI. You can also drag the
74  cyan handles left and right on screen to change which portion of the image is shown as original vs processed.
75 
76  - On JeVois-A33, either try it with JeVois Inventor using <b>YUYV 640x240 @@ 30 fps</b>. Note that each time you
77  select a new \p effect, this will affect the set of parameters that are available for that effect, but currently
78  JeVois Inventor has no way of being notified of that change. So just click to another tab (e.g., the \b Info tab),
79  and then back to the \b Parameters tab each time you change the effect. This will refresh the parameter list.
80 
81  - Or, open a video viewer on your host computer and select <b>YUYV 640x240 @@ 30 fps</b> (see \ref UserQuick)
82 
83  - Open a serial communication to JeVois (see \ref UserCli)
84 
85  - Type `help` and observe the available parameter called \p effect
86 
87  - Start by setting the \p effect parameter to a given effect type. For example:
88  `setpar effect Median` or `setpar effect Morpho` (commands are case-sensitive).
89 
90  - Then type \c help to see what additional parameters are available for each effect. For example, for Median, you
91  can adjust the kernel size (parameter \p ksize). For Morpho, you can select the type of morphological operation
92  (parameter \p op), structuring element shape (parameter \p kshape) and size (parameter \p ksize), etc.
93 
94  Complete example:
95 
96  \code
97  setpar effect Morpho
98  help
99  setpar op Open
100  setpar ksize 7 7
101  \endcode
102 
103  With \jvversion{1.5} and above, you may want to use `help2` instead of `help`, which is a shorter and more compact
104  help message that shows parameters and commands of the running machine vision module only (and no general
105  parameters related to the JeVois core).
106 
107 
108  @author Laurent Itti
109 
110  @videomapping YUYV 640 240 30.0 YUYV 320 240 30.0 JeVois ColorFiltering
111  @email itti\@usc.edu
112  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
113  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
114  @mainurl http://jevois.org
115  @supporturl http://jevois.org/doc
116  @otherurl http://iLab.usc.edu
117  @license GPL v3
118  @distribution Unrestricted
119  @restrictions None
120  \ingroup modules */
122  public jevois::Parameter<effect>
123 {
124  public:
125  // ####################################################################################################
126  //! Inherited constructor ok
127  // ####################################################################################################
129 
130  // ####################################################################################################
131  //! Virtual destructor for safe inheritance
132  // ####################################################################################################
133  virtual ~ColorFiltering()
134  { }
135 
136  // ####################################################################################################
137  //! Processing function with video output to USB
138  // ####################################################################################################
139  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
140  {
141  static jevois::Timer timer("processing", 30, LOG_DEBUG);
142 
143  // Wait for next available camera image:
144  jevois::RawImage inimg = inframe.get(); unsigned int const w = inimg.width, h = inimg.height;
145  inimg.require("input", w, h, V4L2_PIX_FMT_YUYV); // accept any image size but require YUYV pixels
146 
147  timer.start();
148 
149  // While we process it, start a thread to wait for output frame and paste the input image into it:
150  jevois::RawImage outimg; // main thread should not use outimg until paste thread is complete
151  auto paste_fut = jevois::async([&]() {
152  outimg = outframe.get();
153  outimg.require("output", w * 2, h, inimg.fmt);
154  jevois::rawimage::paste(inimg, outimg, 0, 0);
155  jevois::rawimage::writeText(outimg, "JeVois Input Image", 3, 3, jevois::yuyv::White);
156  });
157 
158  // Convert source image to BGR:
159  cv::Mat src = jevois::rawimage::convertToCvBGR(inimg);
160 
161  // Apply the filter (if any):
162  cv::Mat dst; std::string settings;
163  if (itsFilter) settings = itsFilter->process(src, dst);
164  else dst = cv::Mat(h, w, CV_8UC3, cv::Scalar(0,0,0));
165 
166  // Wait for paste to finish up:
167  paste_fut.get();
168 
169  // Paste the results into our output:
170  jevois::rawimage::pasteBGRtoYUYV(dst, outimg, w, 0);
171 
172  std::string const & fpscpu = timer.stop();
173 
174  // Write a few things:
175  jevois::rawimage::writeText(outimg, "JeVois image filter: " + effect::strget(), w + 3, 3, jevois::yuyv::White);
176 
177  std::vector<std::string> svec = jevois::split(settings, "\n");
178  for (size_t i = 0; i < svec.size(); ++i)
179  jevois::rawimage::writeText(outimg, svec[i], w + 3, 15 + 12 * i, jevois::yuyv::White);
180 
181  jevois::rawimage::writeText(outimg, fpscpu, 3, h - 13, jevois::yuyv::White);
182 
183  // Send the output image with our processing results to the host over USB:
184  outframe.send();
185  }
186 
187 #ifdef JEVOIS_PRO
188  // ####################################################################################################
189  //! Processing function with zero-copy and GUI on JeVois-Pro
190  // ####################################################################################################
191  virtual void process(jevois::InputFrame && inframe, jevois::GUIhelper & helper) override
192  {
193  static jevois::Timer timer("processing", 100, LOG_DEBUG);
194  static int left = -1, right = -1;
195 
196  // Start the GUI frame:
197  unsigned short winw, winh;
198  helper.startFrame(winw, winh);
199 
200  // Draw the camera frame:
201  int x = 0, y = 0; unsigned short iw = 0, ih = 0;
202  helper.drawInputFrame("camera", inframe, x, y, iw, ih);
203 
204  helper.itext("JeVois-Pro Color Filtering");
205 
206  timer.start();
207 
208  // Wait for next available camera image:
209  cv::Mat src = inframe.getCvRGBp(); // should be BGR but RGB is faster and should not affect processing...
210  int const sw = src.cols; int const sh = src.rows;
211  inframe.done();
212 
213  // Apply the filter (if any):
214  cv::Mat dst; std::string settings;
215  if (itsFilter) settings = itsFilter->process(src, dst);
216  else dst = cv::Mat(sh, sw, CV_8UC3, cv::Scalar(0,0,0));
217 
218  // We will display the processed image as a partial overlay between two vertical lines defined by 'left' and
219  // 'right', where each also has a mouse drag handle (small square).
220 
221  ImU32 const col = IM_COL32(128,255,255,255); // color of the vertical lines and square handles
222  int const siz = 20; // size of the square handles, in image pixels
223  static bool dragleft = false, dragright = false;
224 
225  // Initialize the handles at 1/4 and 3/4 of image width on first video frame after module is loaded:
226  if (jevois::frameNum() == 0) { left = sw / 4; right = 3 * sw / 4; }
227 
228  // Make sure the handles do not overlap and/or get out of the image bounds:
229  if (left > right - siz) { if (dragright) left = right - siz; else right = left + siz; }
230  left = std::max(siz, std::min(sw - siz * 2, left));
231  right = std::max(left + siz, std::min(sw - siz, right));
232 
233  // Mask and draw the overlay. To achieve this, we convert the whole result image to RGBA and then assign a zero
234  // alpha channel to all pixels to the left of the 'left' bound and to the right of the 'right' bound:
235  cv::Mat ovl; cv::cvtColor(dst, ovl, cv::COLOR_RGB2RGBA);
236  ovl(cv::Rect(0, 0, left, sh)) = 0; // make left side transparent
237  ovl(cv::Rect(right, 0, sw-right, sh)) = 0; // make right side transparent
238  int ox = 0, oy = 0; unsigned short ow = 0, oh = 0;
239  helper.drawImage("filtered", ovl, true, ox, oy, ow, oh, false, true /* overlay */);
240 
241  helper.drawLine(left, 0, left, sh, col);
242  helper.drawRect(left-siz, (sh-siz)/2, left, (sh+siz)/2, col);
243  helper.drawLine(right, 0, right, sh, col);
244  helper.drawRect(right, (sh-siz)/2, right+siz, (sh+siz)/2, col);
245 
246  // Adjust the left and right handles if they get clicked and dragged:
247  ImVec2 const ip = helper.d2i(ImGui::GetMousePos());
248 
249  if (ImGui::IsMouseClicked(0))
250  {
251  // Are we clicking on the left or right handle?
252  if (ip.x > left-siz && ip.x < left && ip.y > (sh-siz)/2 && ip.y < (sh+siz)/2) dragleft = true;
253  if (ip.x > right && ip.x < right+siz && ip.y > (sh-siz)/2 && ip.y < (sh+siz)/2) dragright = true;
254  }
255 
256  if (ImGui::IsMouseDragging(0))
257  {
258  if (dragleft) left = ip.x + 0.5F * siz;
259  if (dragright) right = ip.x - 0.5F * siz;
260  // We will enforce validity of left and right on next frame.
261  }
262 
263  if (ImGui::IsMouseReleased(0))
264  {
265  dragleft = false;
266  dragright = false;
267  }
268 
269  // Write a few things:
270  std::string const & fpscpu = timer.stop();
271  helper.itext("JeVois image filter: " + effect::strget());
272  std::vector<std::string> svec = jevois::split(settings, "\n");
273  for (std::string const & s : svec) helper.itext(s);
274  helper.iinfo(inframe, fpscpu, winw, winh);
275 
276  // Render the image and GUI:
277  helper.endFrame();
278  }
279 #endif
280 
281  // ####################################################################################################
282  protected:
283  //! Parameter callback: set the selected filter algo
284  void onParamChange(effect const & /*param*/, Effect const & val) override
285  {
286  if (itsFilter) { removeSubComponent(itsFilter); itsFilter.reset(); }
287 
288  switch (val)
289  {
290  case Effect::NoEffect: break;
291  case Effect::Blur: itsFilter = addSubComponent<BlurFilter>("filter"); break;
292  case Effect::Median: itsFilter = addSubComponent<MedianFilter>("filter"); break;
293  case Effect::Morpho: itsFilter = addSubComponent<MorphologyFilter>("filter"); break;
294  case Effect::Laplacian: itsFilter = addSubComponent<LaplacianFilter>("filter"); break;
295  case Effect::Bilateral: itsFilter = addSubComponent<BilateralFilter>("filter"); break;
296  }
297  }
298 
299  private:
300  std::shared_ptr<Filter> itsFilter;
301 };
302 
303 // Allow the module to be loaded as a shared object (.so) file:
jevois::GUIhelper
ColorFiltering::process
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function with video output to USB.
Definition: ColorFiltering.C:139
jevois::GUIhelper::drawImage
void drawImage(char const *name, RawImage const &img, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool isoverlay=false)
jevois::OutputFrame
ColorFiltering::~ColorFiltering
virtual ~ColorFiltering()
Virtual destructor for safe inheritance.
Definition: ColorFiltering.C:133
jevois::async
std::future< std::invoke_result_t< std::decay_t< Function >, std::decay_t< Args >... > > async(Function &&f, Args &&... args)
jevois::GUIhelper::startFrame
bool startFrame(unsigned short &w, unsigned short &h)
BlurFilter.H
Timer.H
jevois::GUIhelper::itext
void itext(char const *txt, ImU32 const &col=IM_COL32_BLACK_TRANS, int line=-1)
Module.H
ColorFiltering
Image filtering using OpenCV.
Definition: ColorFiltering.C:121
jevois::split
std::vector< std::string > split(std::string const &input, std::string const &regex="\\s+")
JEVOIS_REGISTER_MODULE
JEVOIS_REGISTER_MODULE(ColorFiltering)
BilateralFilter.H
jevois::GUIhelper::endFrame
void endFrame()
LaplacianFilter.H
jevois::RawImage
jevois::GUIhelper::d2i
ImVec2 d2i(ImVec2 p, char const *name=nullptr)
MorphologyFilter.H
jevois::Timer::start
void start()
jevois::Component::removeSubComponent
void removeSubComponent(std::shared_ptr< Comp > &component)
jevois::rawimage::convertToCvBGR
cv::Mat convertToCvBGR(RawImage const &src)
jevois::ParameterCategory
jevois::GUIhelper::drawRect
void drawRect(float x1, float y1, float x2, float y2, ImU32 col=IM_COL32(128, 255, 128, 255), bool filled=true)
jevois::RawImage::require
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const
jevois::RawImage::width
unsigned int width
jevois::GUIhelper::drawLine
void drawLine(float x1, float y1, float x2, float y2, ImU32 col=IM_COL32(128, 255, 128, 255))
jevois::rawimage::writeText
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
strget
virtual const std::string strget() const=0
ARtoolkit::JEVOIS_DEFINE_ENUM_CLASS
JEVOIS_DEFINE_ENUM_CLASS(Dict,(AR_MATRIX_CODE_3x3)(AR_MATRIX_CODE_3x3_HAMMING63)(AR_MATRIX_CODE_3x3_PARITY65))
Enum for parameter.
Enum.H
jevois::Timer::stop
const std::string & stop(double *seconds)
jevois::Module
MedianFilter.H
RawImageOps.H
jevois::GUIhelper::drawInputFrame
void drawInputFrame(char const *name, InputFrame const &frame, int &x, int &y, unsigned short &w, unsigned short &h, bool noalias=false, bool casync=false)
jevois::RawImage::height
unsigned int height
jevois::GUIhelper::iinfo
void iinfo(jevois::InputFrame const &inframe, std::string const &fpscpu, unsigned short winw=0, unsigned short winh=0)
jevois::InputFrame
jevois::rawimage::pasteBGRtoYUYV
void pasteBGRtoYUYV(cv::Mat const &src, RawImage &dst, int dx, int dy)
jevois::rawimage::paste
void paste(RawImage const &src, RawImage &dest, int dx, int dy)
jevois::RawImage::fmt
unsigned int fmt
h
int h
ColorFiltering::onParamChange
void onParamChange(effect const &, Effect const &val) override
Parameter callback: set the selected filter algo.
Definition: ColorFiltering.C:284
Filter.H
Darknet::JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK
JEVOIS_DECLARE_PARAMETER_WITH_CALLBACK(dataroot, std::string, "Root path for data, config, and weight files. " "If empty, use the module's path.", JEVOIS_SHARE_PATH "/darknet/single", ParamCateg)
Parameter.
demo.w
w
Definition: demo.py:85
jevois::Timer
jevois::Component::Module
friend friend class Module