Optical Camera Communication for Internet of Things in Urban Environments

Abstract

The implementation of Smart Cities might be the path for restraining the critical problems that massive urbanization has caused in the world. These issues deeply impacted the social, economic, and environmental fields. By definition, a smart city is an urban area where data is continuously collected and transmitted for analysis. Powerful analytic tools are applied to these data to extract valuable information for managing assets, resources, and services in the interests of better performance, lower costs, and lessened environmental impact. Therefore, the concept of smarter cities is related to the reduction of energy consumption and greenhouse gas emissions, the inclusion of sustainable transportation systems, and the improvement of human settlements management. Ultimately, the city’s operational efficiency increases, as well as the quality of government services and citizen welfare. The future of smart cities’ applicability relies on today’s research, development, and implementation of Internet of Things (IoT) applications within urban environments. The devices that gather the information can be considered IoT elements that can connect and communicate, through wires or wirelessly, with other apparatus over short, medium, and long distances. However, the massive number of sensors required by a smart city for accurate decision-making limits the implementation of traditional wireless communication links based on Radio-Frequency (RF). This type of communication has two main issues: the spectrum saturation and the inter-systems interference, then additional complementary wireless communication techniques are needed. Some RF-related solutions for Intelligent Transportation Systems (ITS), real-time environmental monitoring, and early disaster detection have been developed already. But those solutions usually interact with a significant amount of interference. Since these data networks are fundamental for future smart cities, these systems’ possible failure is critical. Consequently, their implementation should not depend only on RF links. Considering that social, economic, and environmental problems are global, it is essential to impulse the implementation of smart cities in developed countries, where some pilot cities are already working, and in developing countries where the economic and technological limitations should be considered. Accordingly, some smart cities’ applications require the reuse of previous massive deployments and take advantage of technological trends. In addition, IoT deployments should avoid the saturated spectrum portion, offer reliable communication (redundancy is desirable), moderate delay, low jitter, adequate bit rate, and security. Nowadays, one important technological trend is the use of Light Emitting Diode (LED) for illumination, advertising, and decoration in both indoor and outdoor environments. The LED-based devices present several advantages: energy efficiency, extended life cycles, cost-efficient manufacturing process, and high switching rate. The replacement of fluorescent and halogen lamps by LED illumination systems is part of a global initiative to control excessive power consumption and reduce climate change. This change affects different environments, such as homes, buildings, streets (public lamps), traffic (new signaling lights), and cars (hazard, tail, sidelights, and so forth), even in developing countries. Additionally, there is a movement for using LEDs in walls, clothes, and accessories for decoration purposes and deploying new high-resolution displays based on organic LEDs. This massive utilization of LED-based devices impulses the development and implementation of Visible Light Communication (VLC) systems. VLC is an optical wireless communication technology that works with the visible light portion of the electromagnetic spectrum (430 THz to 790 THz) and transmits data by modulating a light’s characteristic. The emitters are light sources: lamps, screens, or signs. On the reception side, photodiode-based devices are used for sensing the light variation. This technology presents several benefits over RF: larger bandwidth, resistance to RF interference, more security, and an unlicensed spectrum portion. However, the implementation of VLC with photodiodes implies the use of external and potentially expensive devices. For this reason, this study focuses on OCC, a VLC-based technique that uses image sensors for receiving the signals. This technology takes advantage of two trends: the insertion of devices with embedded cameras in a wide variety of day-to-day activities and the massive implementation of outdoor camera-based security and monitoring systems. In general, OCC employs pre-existing devices for transmitting and receiving the signal. Consequently, OCC deployment is low cost. Additionally, smartphones, watches, tablets, and wearables are usually equipped with medium-range resolution cameras which image acquisition speed is acceptable for communication purposes. This technique also includes the possibility of full-duplex communication by using an infrared upload link and the implementation of Multiple-Input Multiple-Output (MIMO) systems by deploying spatially separated light sources. The employment of OCC as a complementary wireless communication technique for deploying IoT applications within urban environments requires an extended analysis. The main goal of the IoT applications on urban environments would be the collection of data. Therefore a unidirectional link could satisfy this objective eliminating the necessity of studying the feasibility of potential uplinks. Similarly, the implementation of MIMO systems and adaptation of image processing techniques to enhance communication can be neglected in this research first stage. However, the study of the communication channel’s characterization and the impact’s analysis of the image sensors characteristics (shutter type, exposition time, distortions) over the data decodification are fundamental for characterizing indoor and outdoor implementations. For applications within urban environments, the distance between the emitters and the receivers can vary from some centimeters to several hectometers, affecting each transmitter’s projection in the acquired image. This issue affects not only the number of pixels in the collected image corresponding to the emitted data source limiting the data transmitted by frame, but it also compromises the meaning of close-emitters interference. To estimate the number of pixels that a light source projects on the image, a set of simple equations were deduced. In this way, the systems can be designed or adapted easily for the minimum quantity of available pixels’ specific constraints. In these links, other light source interference depends on the 2D projection’s proximity, even if the emitters are safely separated in the real world. It is necessary to quantify this interference and then predict its impact over the communication link. For this task, it is fundamental to calculate the number of pixels representing the real-world objects and the relation between the legit signal and the interference, known as Normalized Power Signal to Interference Ratio (NPSIR). Additionally, a study of currently proposed modulation techniques is necessary for developing a long-distance solution. In general, the OOK-based modulations are low complexity techniques that work correctly with moderate data rates exclusively for short distances. Therefore, this type of OCC modulations can be used for specific IoT applications within the range of 5 m, which support a low data rate. Since the polychromatic modulations take advantage of the red, green and blue (RGB) channels for simultaneous transmissions, the data rate increases, however, the achieved distance is short, and the system’s complexity is moderate for Color Shift Modulation (CSK) and high for the CIELUV space. Nevertheless, implementing a polychromatic version of other modulations showed better results regarding data rate without increasing the complexity. The undersampled techniques provide the fastest transmissions rates with moderate to high complexity and medium-range distances. The modulations based on space-time coding reach long distances with relatively low data rates and very high complexity. Since the proposed implementations require specialized hardware for too complex processing, these techniques are not suitable for smart cities in developing countries. Finally, in all the cases, the modulation schemes that compare consecutive frames reach the most extended distances with moderate complexity. Other long-distance modulation schemes that have been proposed rely on significant or high complexity algorithms that require a thresholding technique for identifying the light state. Moreover, the most advanced schemes use bits in each transmission header for calibration purposes, decreasing the throughput. Therefore, for developing a long-distance implementation, modulation schemes based on the comparison of consecutive frames is the best available option that overcomes these constraints. Additionally, the OCC implementations should avoid the flickering problem, so the lights can also be used for illumination, decoration, or advertising purposes without disturbing the people. The flickering free modulations with low complexity algorithms only perform adequately for a medium distance range because the number of bits represented in a single frame decreases with the distance. In general, the available modulation methods work correctly for either short or long distances but not for both cases. In this work, a flickering free, distance independent modulation that overcomes these constraints is proposed. This scheme can be defined as a rolling shutter effect-based modulation that requires a minimum number of light bands for extracting the information. Therefore it can work for short, medium and long distances. Since OCC receivers, image sensors, were designed for obtaining pictures, it is necessary to adapt its operation for recollecting the transmitted data. The first step for achieving this purpose is to analyze a camera’s function. In general, the cameras receive light through an opening in the objective for a defined period, known as exposure time (TEXP). The light is focused on the image sensor chip using the imaging optic lenses. Then the microlenses array focuses the light on the semiconductors for producing electrical signals. Since the photosensors detect light intensity without wavelength specificity, each semiconductor is affected by a color filter, having a one color signal. The collected charge is converted to a voltage that is amplified to produce the electrical signal. A gamma correction tone is applied to the linear photon capture for assuring a normal distribution in the image’s histogram. The electrical signals are digitized, then a spatial interpolation operation (demosaicking) is used for reconstructing a color image. Finally, the image is compressed and stored. The adequate camera’s characteristics can be selected for controlling the image acquisition while highlighting the specific light changes in the frames. The transmitted data must be extracted from the frames. Therefore some digital multilevel signal processing tools are required for preparing the images and videos and bring out the information. Once the frame is processed, the data can be decoded. This research work ”Optical Camera Communication for Internet of Things in Urban Environments” focused on the possibility of implementing IoT applications that transform cities in developing countries by using OCC technology. Since proving the predictions is a fundamental phase in any scientific research, some simulations, lab experiments, and field tests were performed as part of this thesis for validating the possibility of such implementation. At first, a set of trials were done in a darkroom for validating the usability of equations III.28 and III.29, presented in section III.6, for estimating the number of pixels that ureal world objects produce in an image without complex plane transformations or projections. This estimation is fundamental for determining the closeness of light sources within an OCC system; and, therefore, for predicting the interference from close-emitters and its impact on the communication’s performance. In the same way, the emitter’s pixel projection approximation is used as a constrain value during the implementation of the proposed distance-independent modulation scheme. Once the accuracy of this pixels’ estimation was validated, the prediction of close-emitters interference over an OCC communication link had to be proven. For this purpose, initially, an experiment was designed to obtain the values of the relation NPSIR, introduced in section III.7. These experimental measurements were then used to prove the usability of the NPSIR to directly quantify the impact of relative close emitters over the communication’s link performance. An indoor Wireless Sensor Networks using OCC was designed and simulated. The BER of this system was calculated using the NPSIR values. These predictions were compared with the simulation’s outcomes regarding the accuracy of the received signal. In this way, the possibility of estimating the impact of close-emitters interference over an OCC communication link was proved. This prediction of the communication performance’s degradation due to the closeness of light sources can be used for avoiding its effect by applying interference compensation on the receiver side during the data extraction process or by placing the system’s emitters adequately during the applications’ implementation. The next step in the research was to demonstrate the existence of viable OCC applications for IoT. Two proposed implementations were validated with lab tests and field experiments. The first one was an indoor positioning and tracking system, that can be adapted for outdoor applications, while the second one was an outdoor WSN for smart cities. With those simulations and field experiments, the implementation of OCC-based systems for IoT was validated. Nevertheless, the outdoor implementation was no flickering-free; the long-distance demodulation process required the same light state in the full emitter image even if the receiver was a rolling shutter camera. In this particular field experiment, since the selected LED-based device was used for advertising purposes, the flickering was not an issue. However, in general, the outdoor applications need a flickering-free distance-independent modulation scheme, like the one proposed in chapter IV. This process required validation for short, medium, and long-distance rages to be used in different applications. A two-phase procedure was used to prove that this scheme works appropriately. During the first stage, the procedure for selecting the switching frequencies was tested. For this purpose, the modulated signals using the proposed scheme were generated using the switching frequencies obtained by applying equation IV.10 for different frequency multiplier values. Then these signals were used as the input of the light emitters in the frame’s acquisition simulations. From these simulations, the corresponding Pearson correlation coefficient values were calculated to demonstrate the usability of the frequency selection procedure. Once this process was validated, the switching frequencies for three different distance ranges were selected and applied during the second stage. The experimental implementation of the proposed scheme was done using those frequencies for determining the system’s Bit Error Rate and therefore prove that the proposed modulation scheme can be used for applications independently of the distance range. From the simulations, lab evaluations, and field experiments, several conclusions were reached. At first, the usability of equations III.28 and III.29 for estimating the 2D pixel representation of real-world distance was validated with an average pixel error of 0.5605. Additionally, the importance of the minimum focus distance on the pixel representation prediction was established. For the samples taken at a position closer than the minimum focus distance (D 6 40 cm), the maximum error was 2.7547 pixels, while the other samples presented values below one pixel. Furthermore, by taking into account only the pictures that observed that minimum distance, the mean error decreased to 0.3089 pixels (3.55%). The equations III.28 and III.29 estimate the 2D pixel’s projection (x, y) in a photograph of an actual distance (dx, dy) in function of the separation between the object and the camera, the relative angles of the object, and the camera’s characteristics. Therefore the first detailed objective of this thesis was fulfilled. Then the interference from close-emitters within an OCC system was experimentally characterized in function of the transmitted wavelength and then used to calculate the relation NPSIR as a measurement of this chromatic interference. As expected, the worst-case scenario was the transmission of white light, which is affected by the three channels, resulting in lower NPSIR values. Additionally, when the legit emitter’s transmission and the additional light source were done in the same wavelength, or the interference came from a device emitting white light, the NPSIR reaches the lowest results affecting the data communication negatively. For the other samples, the results showed a dependency on the selected wavelength. For the transmissions in blue and red, the interference came from close sources emitting in green, while the LED transmitting in green was affected similarly by blue and red emissions. For the distance D = 100 cm, where the pixel separation between the sources’ boundaries was around six pixels, the NPSIR represented minimum interference, above 86dB, demonstrating that the communication link would not be affected by the close-emitter interference even when transmitting in the same wavelength. Therefore distances above six pixels are considered perfect spatial separation. On the other hand, for the distance D = 200 cm, where the pixel separation between the sources’ boundaries was around one pixel, the NPSIR values for the two sources transmitting in the same wavelength and white light interference were below 50dB. These values would impact the communication link’s performance significantly. Therefore distances below one pixel are considered critical spatial separation. Finally, for the distance D = 140 cm, where the pixel separation between the sources’ boundaries was around three pixels, the NPSIR values for the two sources transmitting in the same wavelength and white light interference were below 75dB. These values would negatively impact the communication link’s performance, but the effect will not be as bad as the critical case. Therefore distances between one and six pixels are considered limited spatial separation. Since the NPSIR measures interference independently of the system characteristics, the experimentally calculated values can be applied for directly estimating the impact of relative close emitters over the communication’s link performance. For this purpose, equation V.4 should be applied with the emitter’s optical power and camera’s Color Filter Array (CFA) and silicon responsivity, as was done in the experiment described in section V.3. The directly calculated BER values were compared with an extensive computational simulation of the camera system over several distances obtaining similar outcomes. Nevertheless, the values from the application of the NPSIR were slightly greater than the other ones in all the cases. Therefore we can conclude that the NPSIR provides an upper limit of the system’s BER. Consequently, the usability of the introduced NPSIR for quantifying the impact of relative close emitters over the OCC systems’ communication link was proved, fulfilling the second detailed objective. The fulfillment of objectives 1 and 2 helped to prove the validity of the hypotheses 1 ”OCC close-emitter interference can be characterized for indoor and outdoor implementations.” The NPSIR provides an estimation of the maximum BER due to the interference of close emitters. This value can be used for predicting the interference of similar systems by projecting the distance in pixels. Therefore, the designed implementations can be validated, and some specific preventive measures can be implemented for mitigating the interference of close emitters. The simulated scenario of section V.3 also proved the feasibility of implementing an indoor WSN based on OCC, contributing to attaining the fourth detailed objective of this thesis. The data communication for the worst case, the sensors one near the other (distance between the sensors’ center is 2.08 cm, corresponding to critical separation), had a maximum simulated BER of 5.09 · 10−4 . When the separation between the centers reaches 4.00 cm (perfect spatial separation) the BER was less than 1 · 10−6 . For proving the validity of the flickering-free distance-independent modulation scheme proposed in chapter IV simulations and field experiments were held. The simulations were developed for testing the procedure for selecting the switching frequencies. Then this procedure was applied for determining the switching frequencies for three different distance ranges. During the field experiments, these switching frequencies were used, and the system’s BER and successful probability were calculated. In this way, the whole proposed modulation scheme was validated for applications independently of the distance range. From the simulations, the resulting frames had at least 18 strip lines for all the tested frequencies. In the other case, the switching frequencies corresponding to α 6 64, the theoretical upper limit for the frequency multiplier, generated bandwidth of at least five pixels that can be easily extracted from longer distances. With these results, the proposed boundaries for the frequency multiplier has been validated. Additionally, the obtained PCC for each data symbol fitted perfectly within the defined range. Therefore, the procedure for selecting the switching frequencies was validated. On the other hand, the field experiment proved the feasibility of implementing the proposed flickering-free distance-independent modulation scheme. For the short distance range, the transmission was successful with BER of 5·10−4 for three consecutive frames at 20 m where 1.5 light bands were extracted. Equivalently, the transmission for medium distance range was successful with BER of 1.8 · 10−3 for 3 consecutive frames and 1.1 · 10−3 for 4 consecutive frames at 40 m where 1.5 light bands were extracted. Finally, at 95 m, long-distance range, the link has BER of 7 · 10−4 for 3 and 4 consecutive frames that contains only 1,5 light bands, and the BER increases to 2 · 10−3 for 3 consecutive frames and up to 7 · 10−4 for 4 consecutive frames at 100 m. Additionally, the success probabilities are above 99.6% for distances that assures at least 1.5 light bands on each frame. These results demonstrated that the modulation scheme is functional, even when only 1.5 light bands are extracted. The proposed modulation method was based on the analysis of the consecutive frames PCC without the implementation of calibration procedures. The simplicity of the wake-up process and the modulation method assures an overall minor complexity, an improvement compared with other long-distance modulation techniques, which presented significant and high complexity. Additionally, the results for three consecutive frames are comparable to the modulation schemes’ outcomes based on the analysis of successive frames (long distance with moderate complexity). At the same time, the throughput of the system (fcam ∗ 2/3 bps) is better. Moreover, the experimental results showed a useful 100 m link using a frequency multiplier below the maximum calculated for the camera’s characteristics. Therefore, longer distances can be easily reached. Since both the simulations and the experiments presented positive results, the flickering-free distance-independent modulation scheme for OCC was validated. Furthermore, the third detailed objective was fulfilled. Since similar BER results and succeed probabilities were obtained for the short, medium, and long-distance range cases, it was proved that the modulation scheme performance does not depend on the distance itself. The real constrain for this modulation technique is the number of light bands extracted from the frame. The system requires at least 1.5 strips for proper demodulation. Therefore, the Hypothesis 2 ”OCC systems can be deployed for medium and long distances by using a distance-independent modulation scheme based on consecutive frames relations without the necessity of previous calibration.” has been validated. The real-time two-step 3D localization system based on OCC was tested under lab conditions. During the experiments, four objects were accurately positioned at the same time. The maximum location error was acceptable for the three dimensions: 3.10 cm in x dimension, 2.65 cm in y dimension, and 1.32 cm in the z dimension. Moreover, the mean location error was 1.2404 cm, with an average processing time of 18.2 ms per frame. These results proved the applicability of such positioning system, which can be easily implemented, for example, in the emerging Industrial Internet of Things (IIoT) applications field for robots’ navigation. The tracking phase of the system was proved with a mean error of 1.07 cm. The proposed method can then be applied for any tracking execution where the objects to be located do not possess embedded cameras but own a LED.based device. Consequently, these experimental results demonstrated that the proposed positioning system contributes to fulfilling this thesis’s fourth detailed objective. It is essential to highlight that the proposed localization system’s position accuracy depends on the camera’s resolution and FoV and the distance to the beacons. Each pixel represents a specific total distance over a plane parallel to the camera’s normal plane in any image. This value is based on the plane and image sensor separation and the camera’s characteristics. If the resolution is increased, the accuracy is also incremented. However, an enhancement on the FoV or a greater separation would decrease the position’s accuracy. For example, each pixel in the photos took during these experiments at a distance of 2.00 m represents 3.40 mm. If the camera’s resolution is changed to 1920 × 1080, each pixel will correspond to 1.24 mm. Therefore, in the second case, the position accuracy will increase remarkably since any on- pixel error would lead to a 1.24 mm location error. Furthermore, the camera’s selection and set up during the system’s implementation should take this effect into account. The outdoor WSN system based on OCC was tested with a field experiment. A long-distance communication link, more than 300 m, between an old generation smartphone and a LED-based advertising device was successfully established. Despite the use of a fixed threshold for decoding the captured signal, the quality of the camera’s sensor, the camera’s instability during the trials, and the environmental conditions (fresh breeze and haze), the system reached an average BER of 1.388 · 10−2 , and a worst-case with BER of 2.222 · 10−2 . Therefore, this experiment demonstrated that an OCC application for IoT within urban environments is possible, fulfilling this thesis’s fourth detailed objective. The experimental results showed that the proposed distance-independent modulation scheme is effective and can be applied for long-distance implementations. Additionally, the feasibility of practical OCC-based implementations for IoT have been demonstrated. Furthermore, for eliminating the flickering issue of the proposed outdoor WSN system, the validated distance-independent modulation scheme can be employed. This WSN implementation can be used for environmental pollution monitoring, temperature or humidity variation control, water contamination monitoring, and disaster detection. Therefore the hypothesis 3 ”OCC is a valid technique for implementing Internet of Things’ systems in Urban Environments” was validated by reaching the detailed objectives three and four.