Projects

This work was achieved in collaboration with Rafael Almar from the LEGOS, Toulouse, France, who provided the video data and reference measures collected during the "2013 Grand Popo experiment"^[1] in Benin.

Concept

In the surf zone, the foam is advected by the sea surface current. Motion estimation techniques enable to estimate the surface current by remotely observing the foam displacement throughout a video sequence. Such measures of the current are useful e.g. to swimming and surfing safety, coastal civil engineering or shore dynamics research.

Here, the main goal was to demonstrate the ability to perform decent measures from a video feed provided by a simple surveillance camera located on the beach (illustration below, scaled down to 25% of original size). To this end, images are first extracted from the video files, then rectified to correct for the perspective view, and filtered to enhance the foam patterns and reduce the visual influence of wave rollers. The figure below shows a sample frame before and after rectification, as well as various areas of interest for the measures. After these pre-processing steps, the Typhoon optical flow (OF) algorithm recovers the 2D surface current. These video-based estimations are finally compared to independent measures provided by an acoustic Doppler velocimeter (ADV).

Results

Comparison of OF measures to that of the ADV proved somewhat difficult. The cross-shore velocity component evolves radically along the vertical direction, and the OF provides surface current whereas the ADV is located below the surface. Nevertheless, measures from these two sources are in remarkably good agreement on both rapid (2 s) and slower (2 min) time scales ^[2]. This study clearly highlights the potential of the method and, given the simplicity of the setup (a basic surveillance camera) and the limited spatial and temporal resolution of video data for this specific OF application, there is much room for improvement.

The video below shows an example of raw surface currents recovered by Typhoon (black vectors) superimposed on rectified input images. The orange vector is the ADV measure, and the teal circle is the area over which OF vectors are averaged to produce time-series. To the right are shown the time-series of velocity: cross-shore component u (top) and along-shore component v (bottom).

References and resources

1. Almar, R., et al. The Grand Popo beach 2013 experiment, Benin, West Africa: from short timescale processes to their integrated impact over long-term coastal evolution. Journal of Coastal Research, 70(sp1), 651-656, 2014.
2. Dérian, P and Almar, R. Wavelet-based Optical Flow Estimation of Instant Surface Currents from Shore-based and UAV Video. IEEE Transactions on Geosciences and Remote Sensing, accepted for publication, 2017.

• project repository on GitHub.
• Transient surf zone circulation induced by rhythmic swash zone at a reflective beach. Coastal Dynamics 2017, Helsingør, Danemark.

Concept

Videos of hydrothermal vents known as "white smokers" are captured from a submarine. The movies show a close view of the "FLO" instrument, a hot-film sensor measuring the fluid flow through a graduated cylinder. Small particles carried by the fluid are also visible. By tracking these particles in the video, it is possible to provide an alternative measure of the fluid velocity and thus estimate the flow.

Demonstrator

Particles are quite large and their density fairly low. This suggests a particle-tracking approach: individual particles are first detected in the region of interest (the cylinder), then each particle is tracked throughout the video sequence. The basic demonstrator simply tracks particles from one frame to the next, while a more advanced implementation would follow each particle from the time it appears in the cylinder until its exit for more robust measures.

Particles are isolated by the "Running Gaussian average" background / foreground detection algorithm. The displacement of each particle is estimated between two consecutive frames using a classical cross-correlation approach. Finally, a quality control process flags and removes potential spurious estimates.

Resources

• whiteSmoker project repository on GitHub (Python).
• Deep-sea environment laboratory at Ifremer (FLO instrument).
• Philippe Rodier, Ifremer (video data).

Concept

Larger scale oceanic models are derived following the "transport under location uncertainty" paradigm. ^[1] Several noise models were designed and implemented in the (stochastic) quasi-geostrophy model SQGMU ^[2] as well as in the NEMO ocean engine. Experiments also involved a stochastic version of the famous "Lorenz 63" model.^[3]

Related works involved a collaboration with Shengze Cai for the design, development and testing of a fluid motion estimator built upon the same "transport under location uncertainty" paradigm.^[4]

References and resources

1. Mémin, E. Fluid flow dynamics under location uncertainty. Geophysical & Astrophysical Fluid Dynamics, 2014
2. Resseguier V., Mémin E. and Chapron B. Geophysical flows under location uncertainty, Part II Quasi-geostrophy and efficient ensemble spreading. Geophysical & Astrophysical Fluid Dynamics, 2014
3. Large scale flows under location uncertainty: a consistent stochastic framework. Quarterly Journal of the Royal Meteorological Society, 2017.
4. Motion Estimation under Location Uncertainty for Turbulent Fluid Flow. Experiments in Fluids, 2018.

• SQGMU-2 project on GitHub.
• Docker-NEMO: Docker image to compile and run the NEMO ocean engine.

Two datasets of scans collected by the SAMPLE aerosol lidar were investigated, including a dataset collected in Chico, CA, alongside the REAL lidar. The later data was used to perform the first direct comparisons of the two instruments,^[1] including wind estimations^[2].

A tool was developed to detect aerosol plumes and export results as KMZ files for integration with Google Earth.

References

1. Comparison of aerosol backscatter and wind field estimates from the REAL and the SAMPLE, SPIE Optical Engineering + Applications, 2015.
2. Two-component wind fields from single scanning aerosol lidar, SPIE Optical Engineering + Applications, 2015.

• Real-time 2D wind estimation from aerosol lidar imagery.
• SAMPLE Winds I (previous project with S3).

This project started with a friend who also faced wind-induced motion in some of his images sequences. While many solutions are readily available for motion compensation, we were curious to explore the underlying mechanisms by ourselves.

Our current implementation (Python) uses correlation-based techniques to identify correspondences within an image pair. These approaches are closely related to some of the motion estimation techniques used for instance on the wind estimation project presented below. Then, the optimal homography matrix, which in this case describes the transformation between two consecutive images that is induced by camera motion, is estimated from the set of correspondences using the RANSAC algorithm.

The video below shows a comparison of the original sequence (left) and the stabilized sequence (right). The stabilization is not perfect, yet the amplitude of camera motion is vastly reduced. The full-length, stabilized movie can be seen in the time-lapse section.

Concept

This project uses the motion of aerosols in the lower atmosphere to infer the underlying horizontal wind field. It is an application of computer vision methods, somewhat similar to the well-known particle image velocimetry (PIV) used in the field of experimental fluid dynamics. Input images are aerosol backscatter data provided by the Raman-shifted Eye-safe Aerosol Lidar (REAL) located in Chico, California.

The apparent displacement of aerosol features between 2 consecutive scans of the REAL are extracted by motion estimation algorithms (illustration below). This displacement field gives us an estimation of the instantaneous wind field.

In the video below, the background is the input backscatter data (copper colors). The estimated motion is visualized by tracers. This example shows the ability to recover a wide range of wind speeds, complex motions, as well as dynamic masking of unreliable estimates in the far range.

Chico 2013 experiment

The Chico 2013 experiment was designed to test the wind estimation in real-time, using two different motion estimation algorithms, and validate the results against a certified instrument.

Chris Mauzey and I designed and implemented the numerical framework that made this experiment possible. It involved the retrieval and organization of raw backscatter data; the motion estimation; as well as visualization of products (e.g. vector fields) in real-time, 24/7 over more than 6 months. Our solution combines Python, C++/CUDA, MySQL and HTML+PHP. Over the course of the experiment, more than 5 TB of raw data were collected and 250 billion wind vectors estimated. The poster below describes some aspects of the experiment (click for full size, 7 MB).

Validation

A commercial doppler lidar, Streamline by Halophotonics, was used for comparisons with our measurements. The doppler lidar was first used in vertical profiling mode (VAD) to compare time-series of 2-component wind vectors at a single location. It was later operated in horizontal scanning mode, alongside the REAL, to compare the line-of-sight velocity component along the range (picture below).

We found an excellent correlation between the doppler measures and our estimates for the 10-min mean wind vector for both algorithms^[1][2].

References

1. Wavelet-based optical flow for two-component wind field estimation from single aerosol lidar data, J. Atm. Oc. Tech., Vol. 32, 2015.
2. Optimization of the cross-correlation algorithm for two-component wind field estimation from single aerosol lidar data and comparison with Doppler lidar, J. Atm. Oc. Tech., Vol. 33, 2016.

• Lidar research group
• Personal project page
• NSF award AGS 1228464

"Processing is an open source programming language and integrated development environment (IDE) built for the electronic arts, new media art, and visual design […]" (Wikipedia). Built upon Java, it enables to design somewhat advanced image processing tasks with live execution capabilities.

In this project, the input video data is read from the webcam (HD 720p). Actors are identified using a background subtraction approach, then replaced by clouds of dark particles subject to gravity. The visual is meant to evocate a mixed of swarming bees, sand whirling, and falling snow as one moves around. The total number of particles is limited (80,000 here) to allow for reasonable processing time and frame rate.

In the example below, the playback was sped up by 10% to compensate for the low input frame rate due to the simultaneous screen recording.

The first step was to identify the specificities of the data collected by the SAMPLE lidar, and develop ad hoc pre-processing solutions. Then, a numerical framework was set up to integrate the Typhoon motion estimation software. Extensive testings and validations were performed on the dataset.

I supervised the whole study, and focused more specifically on the design of the pre-processing steps and the analysis of results. Modifications for the system were proposed in order to optimize the wind estimation capability. The outcome was very successful, both for S3 and the lidar group in Chico. S3 became the first commercial user of Typhoon.