Abstract
Flies are capable of extraordinary flight maneuvers at very high speeds largely due to their highly elaborate visual system. In this work we present a fly-inspired FPGA based sensor system able to visually sense rotations around different body axes, for use on board micro aerial vehicles (MAVs). Rotation sensing is performed analogously to the fly’s VS cell network using zero-crossing detection. An additional key feature of our system is the ease of adding new functionalities akin to the different tasks attributed to the fly’s lobula plate tangential cell network, such as object avoidance or collision detection. Our implementation consists of a modified eneo SC-MVC01 SmartCam module and a custom built circuit board, weighing less than 200 g and consuming less than 4 W while featuring 57,600 individual two-dimensional elementary motion detectors, a 185° field of view and a frame rate of 350 frames per second. This makes our sensor system compact in terms of size, weight and power requirements for easy incorporation into MAV platforms, while autonomously performing all sensing and processing on-board and in real time.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Aubepart F, El Farji M, Franceschini N (2004) FPGA implementation of elementary motion detectors for the visual guidance of micro-air-vehicles. In: Proc IEEE Int Symp Ind Electron, vol 1, pp 71–76
Autrum H (1952) Über zeitliches Auflösungsvermögen and Primärvorgänge im Insektenauge. Naturwissenschaften 39: 290–297
Bermudezi Badia S, Pyk P, Verschure PFM (2007) A fly-locust based neuronal control system applied to an unmanned aerial vehicle: the invertebrate neuronal principles for course stabilization, altitude control and collision avoidance. Int J Robot Res 26: 759–772
Beyeler A, Zufferey J, Floreano D (2009) Vision-based control of near-obstacle flight. Auton Robot 27: 201–219
Borst A (2007) Correlation versus gradient type motion detectors: the pros and cons. Philos Trans R Soc Lond B 362: 369–374
Borst A, Bahde S (1987) Comparison between the movement detection systems underlying the optomotor and the landing response in the housefly. Biol Cybern 56: 217–224
Borst A, Reisenman C, Haag J (2003) Adaptation of response transients in fly motion vision. II: Model studies. Vis Res 43: 1311–1324
Borst A, Haag J, Reiff DF (2010) Fly motion vision. Annu Rev Neurosci 33: 49–70
Buchner E (1976) Elementary movement detectors in an insect visual system. Biol Cybern 24: 85–101
Chiappe ME, Seelig JD, Reiser MB, Jayaraman V (2010) Walking modulates speed sensitivity in Drosophila motion vision. Curr Biol 20: 1470–1475
Conroy J, Gremillion G, Ranganathan B, Humbert JS (2009) Implementation of wide-field integration of optic flow for autonomous quadrotor navigation. Auton Robot 27: 189–198
Cuntz H, Haag J, Forstner F, Segev I, Borst A (2007) Robust coding of flow-field parameters by axo-axonal gap junctions between fly visual interneurons. Proc Natl Acad Sci USA 104:10229–10233
Duistermars BJ, Chow DM, Condro M, Frye MA (2007) The spatial, temporal and contrast properties of expansion and rotation flight optomotor responses in Drosophila. J Exp Biol 210: 3218–3227
Eckert H (1980) Functional properties of the H1-neurone in the third optic ganglion of the blowfly, Phaenicia. J Comp Physiol A 135: 29–39
Egelhaaf M, Borst A, Reichardt W (1989) Computational structure of a biological motion-detection system as revealed by local detector analysis in the fly’s nervous system. J Opt Soc Am A 6: 1070– 1087
Elyada YM, Haag J, Borst A (2009) Different receptive fields in axons and dendrites underlie robust coding in motion-sensitive neurons. Nat Neurosci 12: 327–332
Farrow K, Borst A, Haag J (2005) Sharing receptive fields with your neighbors: tuning the vertical system cells to wide field motion. J Neurosci 25: 3985–3993
Franceschini N, Pichon JM, Blanes C, Brady JM (1992) From insect vision to robot vision. Philos Trans R Soc Lond B 337: 283–294
Franz MO, Krapp HG (2000) Wide-field, motion-sensitive neurons and matched filters for optic flow fields. Biol Cybern 83: 185–197
Fry SN, Rohrseitz N, Straw AD, Dickinson MH (2009) Visual control of flight speed in Drosophila melanogaster. J Exp Biol 212: 1120–1130
Gonzalez RC, Woods RE (2007) Digital image processing, 3rd edn. Prentice Hall, Upper Saddle River, NJ
Goslin M, Mine MR (2004) The Panda3D graphics engine. Computer 37: 112–114
Götz KG, Wenking H (1973) Visual control of locomotion in the walking fruitfly Drosophila. J Comp Physiol A 85: 235–266
Greene N (1986) Environment mapping and other applications of world projections. IEEE Comput Graph Appl 6: 21–29
Haag J, Borst A (2003) Orientation tuning of motion-sensitive neurons shaped by vertical–horizontal network interactions. J Comp Physiol A 189: 363–370
Haag J, Borst A (2004) Neural mechanism underlying complex receptive field properties of motion-sensitive interneurons. Nat Neurosci 7: 628–634
Haag J, Borst A (2005) Dye-coupling visualizes networks of large-field motion-sensitive neurons in the fly. J Comp Physiol A 191: 445–454
Haag J, Borst A (2007) Reciprocal inhibitory connections within a neural network for rotational optic-flow processing. Front Neurosci 1: 111–121
Haag J, Denk W, Borst A (2004) Fly motion vision is based on reichardt detectors regardless of the signal-to-noise ratio. Proc Natl Acad Sci USA 101:16333–16338
Harrison RR (2005) A biologically inspired analog IC for visual collision detection. IEEE Trans Circuits Syst I 52: 2308–2318
Hassenstein B, Reichardt W (1956) Systemtheoretische Analyse der Zeit-, Reihenfolgen- und Vorzeichenauswertung bei der Bewegungsperzeption des Rüsselkäfers Chlorophanus. Z Naturforsch B 11: 513–524
Hausen K, Wehrhahn C (1989) Neural circuits mediating visual flight control in flies. I. Quantitative comparison of neural and behavioral response characteristics. J Neurosci 9: 3828–3836
Hyslop A, Krapp HG, Humbert JS (2010) Control theoretic interpretation of directional motion preferences in optic flow processing interneurons. Biol Cybern 103: 353–364
Joesch M, Plett J, Borst A, Reiff DF (2008) Response properties of motion-sensitive visual interneurons in the lobula plate of Drosophila melanogaster. Curr Biol 18: 368–374
Jung SN, Borst A, Haag J (2011) Flight activity alters velocity tuning of fly motion-sensitive neurons. J Neurosci 31: 9231–9237
Kendoul F, Fantoni I, Nonami K (2009) Optic flow-based vision system for autonomous 3D localization and control of small aerial vehicles. Robot Auton Syst 57: 591–602
Köhler T, Röchter F, Lindemann JP, Möller R (2009) Bio-inspired motion detection in an FPGA-based smart camera module. Bioinspir Biomim 4:015,008 (10 pp)
Krapp HG, Hengstenberg B, Hengstenberg R (1998) Dendritic structure and receptive-field organization of optic flow processing interneurons in the fly. J Neurophysiol 79: 1902–1917
Land MF, Eckert H (1985) Maps of the acute zones of fly eyes. J Comp Physiol A 156: 525–538
Moeckel R, Liu S (2009) Motion detection chips for robotic platforms. In: Floreano D, Zufferey J, Srinivasan MV, Ellington C (eds) Flying insects and robots. Springer, Berlin, pp 101–114
Netter T, Franceschini N (2002) A robotic aircraft that follows terrain using a neuromorphic eye. In: Proceedings of the International Conference on Intellectual Robotic System, vol 1, pp 129–134
O’Carroll DC, Shoemaker PA, Brinkworth RSA (2006) Bio-inspired optical rotation sensor. In: Proceedings of the SPIE Symposium on Smart Structures, Devices, and Systems III, vol 641418, pp 1–12
Petrowitz R, Dahmen H, Egelhaaf M, Krapp HG (2000) Arrangement of optical axes and spatial resolution in the compound eye of the female blowfly Calliphora. J Comp Physiol A 186: 737–746
Poggio T, Reichardt W (1976) Visual control of orientation behaviour in the fly. Part II. Towards the underlying neural interactions. Q Rev Biophys 9: 377–438
Potters M, Bialek W (1994) Statistical mechanics and visual signal processing. J Phys I 4: 1755–1775
Reichardt W, Egelhaaf M (1988) Properties of individual movement detectors as derived from behavioural experiments on the visual system of the fly. Biol Cybern 58: 287–294
Seitz G (1968) Der Strahlengang im Appositionsauge von Calliphora erythrocephala (Meig.). J Comp Physiol A 59(2): 205–231
Single S, Borst A (1998) Dendritic integration and its role in computing image velocity. Science 281: 1848–1850
Srinivasan MV, Chahl JS, Weber K, Venkatesh S, Nagle MG, Zhang SW (1999) Robot navigation inspired by principles of insect vision. Robot Auton Syst 26: 203–216
Srinivasan M, Thurrowgood S, Soccol D (2009) Competent vision and navigation systems. IEEE Robot Autom Mag 16: 59–71
Stumpf P (1911) Über die Abhängigkeit der visuellen Bewegungsrichtung und negativen Nachbildes von den Reizvorgängen auf der Netzhaut. Z Psychol 59: 321–330
Tatler B, O’Carroll DC, Laughlin SB (2000) Temperature and the temporal resolving power of fly photoreceptors. J Comp Physiol A 186: 399–407
Valette F, Ruffier F, Viollet S, Seidl T (2010) Biomimetic optic flow sensing applied to a lunar landing scenario. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp 2253–2260
Weber F, Machens CK, Borst A (2010) Spatiotemporal response properties of optic-flow processing neurons. Neuron 67: 629–642
Wertz A, Gaub B, Plett J, Haag J, Borst A (2009) Robust coding of ego-motion in descending neurons of the fly. J Neurosci 29: 14993–15000
XILINX (1999) XAPP134: Virtex synthesizable high performance SDRAM controller. XILINX
XILINX (2003) XAPP394: Interfacing to mobile SDRAM with CoolRunner-II CPLDs. XILINX
Zhang T, Wu H, Borst A, Kühnlenz K, Buss M (2008) An FPGA implementation of insect-inspired motion detector for high-speed vision systems. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp 335–340
Zufferey J, Floreano D (2006) Fly-inspired visual steering of an ultralight indoor aircraft. IEEE Trans Robot 22: 137–146
Acknowledegments
We wish to thank Väinö Haikala and Hubert Eichner for helpful discussions. This work was supported in part by the Deutsche Forschungsgemeinschaft (DFG) excellence initiative research cluster Cognition for Technical Systems (CoTeSys).
Open Access
This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Plett, J., Bahl, A., Buss, M. et al. Bio-inspired visual ego-rotation sensor for MAVs. Biol Cybern 106, 51–63 (2012). https://doi.org/10.1007/s00422-012-0478-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00422-012-0478-6