Resarch Issue: 3D Interaction

Fachgebiet Augmented Reality (FAR)

General Description

When computer-based information is embedded within the real environment of users, users can react to such information directly within such real-world context - by manipulating real objects in addition to gesturing, speech, 3D pointing, and "standard" human-computer interaction via so-called "WIMP" (desktop-based) user interfaces. We investigate suitable combinations of such interfaces - with particular emphasis on direct 3D human-computer interaction embedded in the real world (tangible UIs).

Associated People

• Permanent Staff: Gudrun Klinker, Björn Schwerdtfeger, Marcus Tönnis
• External Collaborators: Verena Broy (BMW)
• Former Team Members: Christian Sandor

Associated Projects

Chemistry Interaction

More information on the official IGSSE project site

Application Chemistry

ForLog

The focused view on human resources within supra-adaptive logistics systems leads to the fundamentals of / establishes the fundamentals for the mobility of knowledge and the flexibility of employees. (Bayerische Forschungsstiftung (BFS)): The Forlog project

Head Mounted Laser Projector

The Head Mounted Laser Projector is a user centered 3D instruction placement system for industrial applications. The information provided by system distinguishes between where-to-act and what-to-do. The augmented reality laser projector displays the simple where-to-act information directly on an object (e.g. a white body). A standard screen is used to display the more complex what-to-do information. This results in an information presentation suiting industrial needs.

HEART

"Heart surgery Enhanced by Augmented Reality Techniques"; in Cooperation with Deutsches Herzzentrum, Munich.

Minimally invasive or totally endoscopic cardiac surgery is an operation technique in which physicians operate through small incision points at the operating region, either using endoscopic tools or master-slave robot systems. In either case, the field of view of the physician is quite limited to a small area around the operating region. To get more general overview, preoprative imaging and planning data can be used through Augmented Reality techniques.

Demo Heart

Tangible User Interfaces

Interaction techniques for Augmented Reality user interfaces (UIs) differ considerably from well explored 2D UIs, because these include new input and output devices and new interaction metaphors such as tangible interaction. For experimenting with new devices and metaphors we propose a flexible and lightweight UI framework that supports rapid prototyping of multimodal and collaborative UIs. We use the DWARF framework as foundation. It allows us to build highly dynamic systems enabling the exchange of components at runtime.

Our framework is a UI architecture described by a graph of multiple input and output and control components (User Interface Controller UIC).

Associated Publications

	A. Dippon, G. Klinker KinectTouch: Accuracy Test for a Very LowCost 2.5D Multitouch Tracking System ITS 2011, November 13 - 16, Kobe, Japan. (bib)
	P. Maier, A. Dey, C. Waechter, C. Sandor, M. Tönnis, G. Klinker An Empiric Evaluation of Confirmation Methods for Optical See-Through Head-Mounted Display Calibration 10th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, Oct. 26-29, 2011 (bib)
	G. Schubert, E. Artinger, F. Petzold, G. Klinker Tangible tools for architectural design – seamless integration into the architectural workflow Proceedings of ACADIA 2011 (bib)
	E. Artinger, T. Coskun, M. Schanzenbach, F. Echtler, S. Nestler, G. Klinker Exploring Multi-touch Gestures for Map Interaction in Mass Casualty Incidents 3. Workshop zur IT-Unterstützung von Rettungskräften im Rahmen der GI-Jahrestagung Informatik 2011 (bib)
	G. Schubert, E. Artinger, F. Petzold, G. Klinker Bridging the Gap. A (Collaborative) Design Platform for early design stages Proceedings of eCAADe 2011 (bib)
	A. Dippon, F. Echtler, G. Klinker Multi-Touch Table as Conventional Input Device HCI International Extended Abstracts, July 9 - 14, Orlando, Florida, USA, pp. 237-241. (bib)
	M. Tönnis, A. Benzina, G. Klinker Utilizing Consumer 3D TV Hardware for a Flexibly Reconfigurable Visualization System Technical Report (TUM-I1113), Technische Universität München, Germany, Jul. 1, 2011. (bib)
	M. Tönnis, D. A. Plecher Presentation Principles in Augmented Reality - Classification and Categorization Guidelines (Version 1.0) Technical Report (TUM-I1111), Technische Universität München, Germany, Jun. 6, 2011. (bib)
	N. Klügel, M. Frieß, G. Groh, F. Echtler An Approach to Collaborative Music Composition International Conference on New Interfaces for Musical Expression (NIME 2011), May 30th, 2011, Oslo, Norway. (bib)
	A. Benzina, M. Tönnis, M. Ashry, G. Klinker Phone-Based Motion Control in VR - Analysis of Degrees of Freedom CHI Extended Abstracts'2011, Vancouver, Canada, May 7 - 12, 2011, pp. 1519-1524. (bib)
	F. Echtler, G. Klinker, A. Butz Towards a Unified Gesture Description Language 13th International Conference on Humans and Computers (HC 2010), December 8, 2010, Düsseldorf, Germany. (bib)
	P. Maier, M. Tönnis, G. Klinker Designing and Comparing Two-Handed Gestures to Confirm Links between User Controlled Objects 9th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), Seoul, Korea, Oct. 13-16, 2010 (bib)
	E. Artinger, M. Schanzenbach, F. Echtler, S. Nestler, T. Coskun, G. Klinker Beyond Pinch-to-Zoom: Exploring Alternative Multi-touch Gestures for Map Interaction Technischer Bericht: TUM-I1006 (bib)
	S. Nestler, T. Coskun, E. Artinger, P. Pichlmaier, G. Klinker Indirect Tracking of Patients in Mass Casualty Incidents 2. Workshop zur IT-Unterstützung von Rettungskräften im Rahmen der GI-Jahrestagung Informatik 2010 (bib)
	T. Coskun, S. Nestler, E. Artinger, A. Benzina, G. Klinker Is it possible to interact with a handheld device while holding it in both hands? 2. Workshop zur IT-Unterstützung von Rettungskräften im Rahmen der GI-Jahrestagung Informatik 2010 (bib)
	A. Dippon Multi-Touch Devices as Conventional Input Devices Diploma Thesis 2010. (bib)
	S. Nestler, E. Artinger, T. Coskun, Y. Yildirim-Krannig, S. Schumann, M. Maehler, F. Wucholt, S. Strohschneider, G. Klinker Assessing Qualitative Usability in life-threatening, time-critical and unstable Situations 10. Workshop Mobile Informationstechnologien / Mobiles Computing in der Medizin (MoCoMed 2010) (bib)
	S. Nestler, E. Artinger, T. Coskun, T. Endres, G. Klinker RFID based Patient Registration in Mass Casualty Incidents 10. Workshop Mobile Informationstechnologien / Mobiles Computing in der Medizin (MoCoMed 2010) (bib)
	T. Coskun, E. Artinger, S. Nestler, A. Benzina, G. Klinker Tablet device interaction with geographical information in time-critical, life- threatening and unstable situations 10.Workshop Mobile Informationstechnologien in der Medizin (Mocomed 2010) (bib)
	S. Nestler Design, Implementation and Evaluation of User-Interfaces for life-threatening, time-critical and unstable Situations Ph.D. Thesis, Technische Universität München (bib)
	M. Frieß, M. Kleinhans, F. Forster, F. Echtler, G. Groh A Tabletop Interface for Generic Creativity Techniques International Conference on Interfaces and Human Computer Interaction (IHCI 2010), July 26, 2011, Freiburg, Germany. (bib)
	F. Echtler, G. Klinker, A. Butz Features, Regions, Gestures: Components of a Generic Gesture Recognition Engine Workshop on Engineering Patterns for Multitouch Interfaces, June 20, 2010, Berlin, Germany. (bib)
	F. Echtler, M. Häussler, G. Klinker BioTISCH: the interactive molecular biology lab bench CHI 2010 "Works-in-Progress", April 10 - 14, Atlanta, GA, USA. (bib)
	P. Maier, M. Tönnis, G. Klinker, A. Raith, M. Drees, F. Kühn What Do You Do When Two Hands Are Not Enough? Interactive Selection of Bonds Between Pairs of Tangible Molecules 5th International IEEE Symposium on 3D User Interfaces, Waltham, MA, March 2010, pp. 83-90. (bib)
	A. Dippon Development of Multitouch-Enabled Games SEP Thesis 2010. (bib)
	F. Echtler, T. Pototschnig, G. Klinker An LED-based Multitouch Sensor for LCD Screens TEI 2010, January 25 - 27, Cambridge, MA, USA. (bib)
	F. Echtler, A. Dippon, M. Tönnis, G. Klinker Inverted FTIR: Easy Multitouch Sensing for Flatscreens ITS 2009, November 23 - 25, Banff, Canada. (bib)
	F. Echtler Tangible Information Displays Dissertation an der Fakultät für Informatik, Technische Universität München, 2009 (online version available here) (bib)
	S. Nestler, M. Huber, G. Klinker Improving the documentation in mass casualty incidents by combining electronic and paper based approaches AmI-09, Roots for the Future of Ambient Intelligence, Salzburg, 18th - 21st Nov 2009 (bib)
	S. Nestler, M. Huber, G. Klinker Hybrid approach for management of patient-related information in mass casualty incidents Technical Report, TUM-I0926 (bib)
	P. Maier Augmented Chemical Reactions First International Workshop on Computational Engineering - Special Topic Fluid-Structure Interaction (bib)
	F. Echtler, S. Nestler, A. Dippon, G. Klinker Supporting Casual Interactions between Board Games on Public Tabletop Displays and Mobile Devices Personal and Ubiquitous Computing, Special Issue on Public and Private Displays, Springer Verlag, 2009, pp. 609-617. (bib)
	S. Nestler, G. Klinker Mobile computing in mass casualty incidents (MCIs) Mobiles Computing in der Medizin (MoCoMed), Lübeck, 29. September 2009 (bib)
	P. Maier, M. Tönnis, G. Klinker Dynamics in Tangible Chemical Reactions International Conference on Chemical Engineering (ICCE 2009), Amsterdam, Netherlands, Sep 23 - 25, 2009 (bib)
	D. Popiv, M. Rakic, K.-J. Bengler, H.Bubb, S. Nestler Timing Concept for Assistance of Anticipatory Driving 17th World Congress on Ergonomics (IEA 2009) (bib)
	S. Nestler, M. Duschl, D. Popiv, M. Rakic, G. Klinker Concept for visualizing concealed objects to improve the driver’s anticipation 17th World Congress on Ergonomics (IEA 2009) (bib)
	A. Dippon Gleichzeitige Ansteuerung einer LED-Matrix als Display und Sensor IDP Thesis in Electrical Engineering 2009. (bib)
	S. Nestler, G. Klinker Mobile User-Interfaces for Text Input in Time-Critical, Unstable and Life-Threatening Situations HCI International 2009 - Posters (bib)
	S. Nestler, M. Tönnis, G. Klinker Common Interaction Schemes for In-Vehicle User-Interfaces Human-Computer Interaction, Lecture Notes in Computer Science, Volume 5612 (bib)
	M. Tönnis, F. Echtler, M. Huber, G. Klinker Low Cost 3D Rotational Input Devices: the stationary Spinball and the mobile Soap3D The 11th Symposium on Virtual and Augmented Reality (SVR), Porto Alegre, Brazil, May 25 - 28, 2009. Best short paper (bib)
	P. Maier, M. Tönnis, G. Klinker Augmented Reality for teaching spatial relations IJAS American Canadian Conference for Academic Disciplines, Toronto, Canada, May 25 - 28, 2009 (bib)
	J. Schöning, J. Hook, N. Motamedi, P. Olivier, F. Echtler, P. Brandl, L. Muller, F. Daiber, O. Hilliges, M. Löchtefeld, T. Roth, U. von Zadow Building Interactive Multi-Touch Surfaces Journal of Graphics, GPU, & Game Tools (bib)
	F. Echtler, T. Sielhorst, M. Huber, G. Klinker A Short Guide to Modulated Light TEI 2009, February 16 - 18, Cambridge, United Kingdom. (bib)
	S. Nestler, M. Huber, F. Echtler, A. Dollinger, G. Klinker Development and evaluation of a virtual reality patient simulation (VRPS) The 17-th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG'09) (bib)
	G. Klinker SudokuVis - How to Explore Relationships of Mutual Exclusion 4th International Symposium on Visual Computing (ISVC), Las Vegas, Nevada, USA, December 2008. (bib)
	F. Echtler, G. Klinker A Multitouch Software Architecture Proc. of the 5th Nordic Conference on Human-Computer Interaction: Using Bridges (NordiCHI? 08), Lund, Schweden, pp. 463-466. (bib)
	J. Schöning, P. Brandl, F. Daiber, F. Echtler, O. Hilliges, J. Hook, M. Löchtefeld, N. Motamedi, L. Muller, P. Olivier, T. Roth, U. von Zadow Multi-Touch Surfaces: A Technical Guide Technical Report TUM-I0833 (bib)
	P. Maier Demo - Augmented Chemical Reactions - Small molecules with a large impact The 7th IEEE and ACM International Symposium on Mixed and Augmented Reality, Cambridge, UK, Sept. 15 - 18, 2008 (bib)
	F. Echtler, G. Klinker Tracking Mobile Phones on Interactive Tabletops MEIS '08: Workshop on Mobile and Embedded Interactive Systems, Munich, Germany (bib)
	P. Maier, G. Klinker, M. Drees, F. Kühn Tangible Chemical Reactions - Small molecules with a large impact IGSSE Forum 2008, Raitenhashach, Germany, June 27 - 28, 2008 (bib)
	S. Nestler, F. Echtler, A. Dippon, G. Klinker Collaborative problem solving on mobile hand-held devices and stationary multi-touch interfaces PPD'08. Workshop on designing multi-touch interaction techniques for coupled public and private displays (bib)
	F. Echtler, M. Huber, G. Klinker Shadow Tracking on Multi-Touch Tables AVI '08: 9th International Working Conference on Advanced Visual Interfaces, Naples, Italy (bib)
	S. Nestler Einsatzorganisation in Katastrophen. Unterstützung von Einsatzkräften durch mobile User Interfaces VDM Verlag Dr. Müller (bib)
	M. Tönnis The Tangible Car - Rapid Intuitive Traffic Scenario Generation in a Hybrid Table-top and Virtual Environment The Fourth International Workshop on the Tangible Space Initiative in conjuntion with the Sixth IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan, Nov. 13 - 16, 2007. (bib)
	M. Tönnis, M. Huber Ambient Interfaces - Ubiquitous Tracking and the Sphere of Vision The International Young Researchers' Workshop on Ambient Interfaces, Osaka, Japan, Nov. 11 - 12, 2007. (bib)
	F. Echtler, M. Huber, G. Klinker Hand tracking for enhanced gesture recognition on interactive multi-touch surfaces Technical Report TUM-I0721, Nov. 1, 2007. (bib)
	S. Nestler, G. Klinker Towards adaptive user-interfaces: Developing mobile user-interfaces for the health care domain Mobiles Computing in der Medizin (MoCoMed), Augsburg, 20. September 2007 (bib)
	B. Schwerdtfeger, M. Kellerer, G. Klinker Munich Hunt: Playing cops and robbers all over Munich 9th International Conference on Ubiquitous Computing (UbiComp07?), Insbruck, Austria, September 16-19, 2007. (bib)
	S. Nestler, G. Klinker Using Mobile Hand-Held Computers in Disasters UbiComp Workshop on Interaction with Ubiquitous Wellness and Healthcare Applications (UbiWell), Innsbruck, 16. September 2007 (bib)
	S. Nestler, A. Dollinger, F. Echtler, M. Huber, G. Klinker Design and Development of Virtual Patients Vierter Workshop Virtuelle und Erweiterte Realität der GI-Fachgruppe VR/AR, Weimar, 15. Juli 2007 (bib)
	B. Schwerdtfeger, T. Frimor, D. Pustka, G. Klinker Mobile Information Presentation Schemes for Logistics Applications 16th International Conference on Artificial Reality and Telexistence (ICAT 2006), Hangzhou, P.R.China, Nov. 29 - Dec. 1, 2006. (bib)
	S. Nölle, G. Klinker Augmented Reality as a Comparison Tool in Automotive Industry The Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality, Santa Barbara, CA, USA, Oct. 22 - 25, 2006. (bib)
	G. Hillebrand, M. Bauer, K. Achatz, G. Klinker Inverse Kinematic Infrared Optical Finger Tracking 9th International Conference on Humans and Computers (HC 2006), Aizu, Japan (bib)
	S. Nestler Resource management in disasters Diploma Thesis, TU München, 2006 (bib)
	M. Tönnis, V. Broy, G. Klinker A Survey of Challenges Related to the Design of 3D User Interfaces for Car Drivers The 1st IEEE Symposium on 3D User Interfaces (in conjunction with IEEE Virtual Reality Conference), Alexandria, Virginia USA, Mar. 25 - 26, 2006, pp. 127-134. (bib)
	O. Hilliges, C. Sandor, G. Klinker Interactive Prototyping for Ubiquitous Augmented Reality User Interfaces International Conference on Intelligent User Interfaces (IUI), January 29 - February 1, 2006, Sydney, Australia (bib)
	C. Sandor, G. Klinker Lessons Learned in Designing Ubiquitous Augmented Reality User Interfaces Chapter in upcoming book: "Emerging Technologies of Augmented Reality: Interfaces & Design" Michael Haller, Mark Billinghurst, Bruce Thomas, editors. 2006. Idea group inc. publishers (bib)
	C. Sandor A Software Toolkit and Authoring Tools for User Interfaces in Ubiquitous Augmented Reality PHD Thesis (bib)
	C. Sandor, A. Olwal, B. Bell, S. Feiner Immersive Mixed-Reality Configuration of Hybrid User Interfaces The Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality, Vienna, Austria, Oct. 5 - 8, 2005. (bib)
	P. Georgel, C. Sandor, M. Meyer, R. Whitaker, G. Klinker A Novel Approach to Automatic Layout for User Interface Elements in Augmented Reality Demo at the Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality, Vienna, Austria, Oct. 5 - 8, 2005. (bib)
	J. Georgii, F. Echtler, R. Westermann Interactive Simulation of Deformable Bodies on GPUs Simulation and Visualisation Conference Proceedings, 2005 (bib)
	C. Sandor, G. Klinker A Rapid Prototyping Software Infrastructure for User Interfaces in Ubiquitous Augmented Reality Journal for Personal and Ubiquitous Computing, Springer Verlag, 2005, pp. 169-185. The original publication is available online at www.springerlink.com . (bib)
	C. Sandor, B. Bell, A. Olwal, N. Temiyabutr, S. Feiner Visual End User Configuration of Hybrid User Interfaces Demo at International Symposium on Mixed and Augmented Reality (ISMAR) 2004, Arlington, VA, USA, Nov 2-4, 2004 (bib)
	V. Novak, C. Sandor, G. Klinker An AR Workbench for Experimenting with Attentive User Interfaces International Symposium on Mixed and Augmented Reality (ISMAR) 2004, Arlington, VA, USA, Nov 2-4, 2004 (bib)
	C. Sandor, B. Bell, A. Olwal, N. Temiyabutr, S. Feiner Visual End User Configuration of Hybrid User Interfaces Demo description in ACM SIGMM 2004 Workshop on Effective Telepresence, New York, NY, USA, Oct. 15, 2004. (bib)
	R. Bauernschmitt, M. Feuerstein, E.U. Schirmbeck, J. Traub, G. Klinker, S.M. Wildhirt, R. Lange Improved Preoperative Planning in Robotic Heart Surgery IEEE Proceedings of Computers in Cardiology (CinC 2004), pp. 773 - 776, Chicago, USA, September 2004 (bib)
	J. Traub, M. Feuerstein, M. Bauer, E.U. Schirmbeck, H. Najafi, R. Bauernschmitt, G. Klinker Augmented Reality for Port Placement and Navigation in Robotically Assisted Minimally Invasive Cardiovascular Surgery Proceedings of Computer Assisted Radiology and Surgery (CARS 2004), pp. 735 - 740, Chicago, USA, June 2004 (bib)
	R. Bauernschmitt, E.U. Schirmbeck, M. Groher, P. Keitler, M. Bauer, H. Najafi, G. Klinker, R. Lange Navigierte Platzierung endovaskulärer Aortenstents Z Kardiologie 2004 93 S3: 116 (bib)
	C. Kulas, C. Sandor, G. Klinker Towards a Development Methodology for Augmented Reality User Interfaces Proc. of the International Workshop exploring the Design and Engineering of Mixed Reality Systems - MIXER 2004, Funchal, Madeira, CEUR Workshop Proceedings (bib)
	O. Hilliges, C. Sandor, G. Klinker A Lightweight Approach for Experimenting with Tangible Interaction Metaphors Proc. of the International Workshop on Multi-user and Ubiquitous User Interfaces (MU3I) 2004, Funchal, Madeira (bib)
	A. MacWilliams, C. Sandor, M. Wagner, M. Bauer, G. Klinker, B. Bruegge Herding Sheep: Live System Development for Distributed Augmented Reality Proceedings of The Second International Symposium on Mixed and Augmented Reality (ISMAR 2003) (bib)
	C. Sandor, A. MacWilliams, M. Wagner, M. Bauer, G. Klinker SHEEP: The Shared Environment Entertainment Pasture Demo at IEEE and ACM International Symposium on Mixed and Augmented Reality ISMAR 2002, Darmstadt, Germany, Sept. 30 - Oct. 1, 2002. (bib)
	C. Sandor, T. Reicher CUIML: A Language for Generating Multimodal Human-Computer Interfaces Proceedings of the European UIML Conference, 2001 (bib)
	G. Klinker, D. Stricker, D. Reiners An Optically Based Direct Manipulation Interface for Human-Computer Interaction in an Augmented World Computers and Graphics 23(6), 1999. (Also in Proc. Workshop EGVE'99, Austria, 1999.) (bib)
	G. Klinker, D. Stricker, D. Reiners Augmented Reality: A Balance Act between High Quality and Real-Time Constraints Proc. 1rst International Symposium on Mixed Reality (ISMR'99): Y. Ohta and H. Tamura (eds.), "Mixed Reality - Merging Real and Virtual Worlds", 1999. (bib)
	G. Klinker, K. Ahlers, D. Breen, P.-Y. Chevalier, C. Crampton, D. Greer, A. Kramer, E. Rose, M. Tuceryan, R. Whitaker Confluence of Computer Vision and Interactive Graphics for Augmented Reality PRESENCE - Teleoperators and Virtual Environments, Special Issue on Augmented Reality 6(4), 1997, pp. 433-451. (bib)