JeVoisBase  1.5
JeVois Smart Embedded Machine Vision Toolkit Base Modules
Share this page:
DemoEyeTracker.C
Go to the documentation of this file.
1 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
2 //
3 // JeVois Smart Embedded Machine Vision Toolkit - Copyright (C) 2016 by Laurent Itti, the University of Southern
4 // California (USC), and iLab at USC. See http://iLab.usc.edu and http://jevois.org for information about this project.
5 //
6 // This file is part of the JeVois Smart Embedded Machine Vision Toolkit. This program is free software; you can
7 // redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software
8 // Foundation, version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
9 // without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
10 // License for more details. You should have received a copy of the GNU General Public License along with this program;
11 // if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
12 //
13 // Contact information: Laurent Itti - 3641 Watt Way, HNB-07A - Los Angeles, CA 90089-2520 - USA.
14 // Tel: +1 213 740 3527 - itti@pollux.usc.edu - http://iLab.usc.edu - http://jevois.org
15 // ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
16 /*! \file */
17 
18 #include <jevois/Core/Module.H>
21 #include <jevois/Debug/Timer.H>
22 
23 #include <linux/videodev2.h> // for v4l2 pixel types
24 #include <opencv2/imgproc/imgproc.hpp>
25 
26 // icon by Icomoon in medical at flaticon
27 
28 //! Pupil detection and eye-tracker using the openEyes toolkit
29 /*! This module implements an eye tracker, which is based on detecting the outline of the pupil.
30 
31  Note that the camera has to be very close to the eye for this to work well. To be useful in practice, some sort of
32  prism or tele-lens should be used so that the camera can be out of the field of view of the human participant.
33 
34  This module can easily run at 120 frames/s.
35 
36  The original eye tracking software used here can be found at http://thirtysixthspan.com/openEyes/software.html
37 
38  This module only performs the required image processing. To build a complete eye tracking system, additional
39  software is required (which can be provided by several open-source eye-tracking toolkits):
40 
41  - Establish a calibration procedure, whereby the participant will be asked to look at small dots at known locations
42  on the computer screen. This will allow a comparison between coordinates of the pupil center as reported by JeVois
43  and coordinates on the computer screen.
44 
45  - Compute the calibration transform (mapping from pupil coordinates reported by JeVois to screen coordinates) based
46  on the data obtained by the calibration procedure. This has to be done once for each participant and eye-tracking
47  session, or every time a participant moves her/his head by a large amount.
48 
49  - On new eye-tracking sessions, convert pupil coordinates reported by JeVois to screen coordinates.
50 
51  - If desired, extract events in the raw calibrated data stream, such as fixations, rapid saccadic eye movements,
52  smooth pursuit movements, and blinks.
53 
54 
55  @author Laurent Itti
56 
57  @videomapping GREY 640 480 30.0 YUYV 640 480 30.0 JeVois DemoEyeTracker
58  @videomapping GREY 320 240 60.0 YUYV 320 240 60.0 JeVois DemoEyeTracker
59  @videomapping GREY 176 144 120.0 YUYV 176 144 120.0 JeVois DemoEyeTracker
60  @email itti\@usc.edu
61  @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
62  @copyright Copyright (C) 2016 by Laurent Itti, iLab and the University of Southern California
63  @mainurl http://jevois.org
64  @supporturl http://jevois.org/doc
65  @otherurl http://iLab.usc.edu
66  @license GPL v3
67  @distribution Unrestricted
68  @restrictions None
69  \ingroup modules */
71 {
72  public:
73 
74  //! Constructor
75  DemoEyeTracker(std::string const & instance) : jevois::Module(instance)
76  { itsEyeTracker = addSubComponent<EyeTracker>("eyetracker"); }
77 
78  //! Virtual destructor for safe inheritance
79  virtual ~DemoEyeTracker() { }
80 
81  //! Processing function
82  virtual void process(jevois::InputFrame && inframe, jevois::OutputFrame && outframe) override
83  {
84  static jevois::Timer timer("processing");
85  static double pupell[5]; // pupil ellipse data
86 
87  // Wait for next available camera image:
88  jevois::RawImage const inimg = inframe.get(true);
89 
90  // Wait for an image from our gadget driver into which we will put our results:
91  jevois::RawImage outimg = outframe.get();
92 
93  // Enforce that the input and output dims match, input should be YUYV, output gray:
94  inimg.require("input", outimg.width, outimg.height, V4L2_PIX_FMT_YUYV);
95  outimg.require("output", inimg.width, inimg.height, V4L2_PIX_FMT_GREY);
96 
97  timer.start();
98 
99  // Convert the input to grayscale, directly into the output's pixel buffer:
100  cv::Mat cvin = jevois::rawimage::cvImage(inimg);
101  cv::Mat cvout = jevois::rawimage::cvImage(outimg);
102  cv::cvtColor(cvin, cvout, CV_YUV2GRAY_YUYV);
103 
104  // Let camera know we are done processing the input image:
105  inframe.done();
106 
107  // Process through the eye tracker, it wil draw a bunch of things into the output image:
108  itsEyeTracker->process(cvout, pupell);
109 
110  // Show processing fps:
111  std::string const & fpscpu = timer.stop();
112  jevois::rawimage::writeText(outimg, fpscpu, 3, outimg.height - 13, 255);
113  jevois::rawimage::writeText(outimg, "JeVois EyeTracker", 3, 3, 255);
114 
115  // Send the output image with our processing results to the host over USB:
116  outframe.send();
117  }
118 
119  protected:
120  std::shared_ptr<EyeTracker> itsEyeTracker;
121 };
122 
123 // Allow the module to be loaded as a shared object (.so) file:
cv::Mat cvImage(RawImage const &src)
friend friend class Module
virtual void process(jevois::InputFrame &&inframe, jevois::OutputFrame &&outframe) override
Processing function.
void writeText(RawImage &img, std::string const &txt, int x, int y, unsigned int col, Font font=Font6x10)
unsigned int height
virtual ~DemoEyeTracker()
Virtual destructor for safe inheritance.
DemoEyeTracker(std::string const &instance)
Constructor.
std::string const & stop()
Pupil detection and eye-tracker using the openEyes toolkit.
JEVOIS_REGISTER_MODULE(DemoEyeTracker)
std::shared_ptr< EyeTracker > itsEyeTracker
unsigned int width
void require(char const *info, unsigned int w, unsigned int h, unsigned int f) const