We built GestureWorks to speed the development of our own multitouch and multiuser exhibits, but soon realized it this would be of value to others Flash developers. 

GestureWorks provides a single solution for the HP TouchSmart and Dell Studio One multitouch enabled computers. We include optimized drivers for the NextWindow (multitouch overlay) that comes with each of these systems. The framework  greatly speeds development in Flash.  It is also extremely efficient: multitouch enabled applications created with GestureWorks run faster, using significantly less CPU than those using other authoring approaches.  GestureWorks will be available for Mac OS X and Windows and will work with Adobe Flash CS3 and CS4.

Additionally, GestureWorks will ship free with our multitouch table, and it works in conjunction with NUI’s Snowflake software. You can see a full list of features and performance information on the GestureWorks site.

To be notified of the free public Alpha, you can join the mailing list on the GestureWorks site. You can also follow GestureWorks on Twitter.

I’ll post more pictures as the new space is built out.

As a quick proof of concept, we downloaded the WiiFlash Server for Mac and took a look at some the examples provided on the WiiFlash site. 

In no time, Jonathan here at the studio had some examples working with the Wii controller and the Wii balance board. The controller examples took advantage of the motion sensing built into the device. (You can learn more about how the Wii works at the NY Times website. ) As you move the Wii controller a 3D-image of plane rotates and moves in unison on the screen.

We also tried out DarwiinRemote which turns the Wii infrared sensors into mouse coordinates. Both this application and the WiiFlash server connect via a bluetooth device in your computer. Any mouse or keyboard action can be mapped to the Wiimote buttons.

We also came across examples that use the controller as “receiver” with LEDs are used as input devices. A good example of this application is WiiSpray. 

Securing the Wii controller in a museum environment is a major concern, as is power to the controller. Still, one could imagine providing constant power through some sort of tether that might simultaneously secure the device.

The Wii balance board shows a lot promise for museum exhibits. It provides a simple way to measure a visitor’s weight and get that data into the computer. One could easily picture a “your weight on other worlds” exhibit (see a simple online version at the Exploratorium). The board and Flash can also be used to detect the weight of each quadrant along with the total weight being registered on the board.  Side-to-side and front-and-back movement can be detected via the four quadrants along Flash to detect shifts in weight.

The Wii Balance Board.

The output in WiiFlash displaying weight from the different quadrants.

The WiiFlash demo showing the total weight in Kilograms.

One limitation is the WiiFlash server cannot read the current battery power of the board. However, in a museum exhibit you’d need to wire direct power to battery area anyway. You’d also need to find a way to secure the board. The only major limitation to using the Wii balance board in a museum environment is that it needs to detect the blue tooth connection with the computer on start-up.  This requires pressing a button on the bottom of the board when the computer boots. So, constant power would be necessary. Of course, this not a very “green” option.

We’ll post more about the Wii and Flash as a potential exhibit development platform as we continue to experiment.

Beginning in July, we are offering an affordable off-the-shelf kiosk stand that will accommodate any size HP TouchSmart, and VESA compliant LCD monitors up to 42.”

The rugged Ideum kiosk stand is made for professional installations in high-traffic environments. It is machined from heavy-gauge powder coated steel, with a dual -reinforced integrated VESA bracket, and precision high strength welding. It features an adjustable or fixed screen angle, and an ADA compliant adjustable height.

The stand incorporates a hidden cable run, with openings large enough to accommodate VGA or DVI cables, vibration dampening, pinhole power button access, and a small 18″ x 22″ footprint – designed for bolting to the floor.

The Ideum touch smart stand provides an elegant and affordable kiosk solution for the HP TouchSmart. Retail pricing will be around $1300, to be announced when we begin shipping in July.

For full size images of the stand visit our Flickr page.

The 100″ multitouch table will use two high-resolution projectors, yielding a combined resolution of 2500 x 720. The two projected images will be blended together using a specialized video card producing a seamless 7:2 ratio, ultra-wide screen image. 

The table design has been modified to include a contiguous glass surface with rounded edges and wrap around powder-coated black panels. Underneath, bright blue LED lights provide “ground effects.” We’ve been working with Adventure Science Center and our design partners at Ralph Applebaum and Associates to finalize the look of the 100″ table.

The exhibit will be multitouch and multiuser. The table will present a representation of the electromagnetic spectrum from Radio to Gamma-Rays. The visible spectrum appears in the center of the table.

Visitors (up to 8 or more simultaneously) will be able interact with images of terrestrial and celestial  objects that appear on the table surface. Visitors can rotate, scale and move the images across the table. As images pass from one wavelength to another, a new image is revealed showing the object as it appears in the new wavelength.  Contextual metadata will allow visitors to learn more about the image(s) they are interacting with.

The exhibit can be approached from either side.  Wavelength “drawers” can be popped-out revealing more information about the wavelength the visitor is exploring.

We’re using Natural User Interface’s Snowflake software for optical support and for “blob tracking.” The exhibit uses two computers.  One will do blob tracking, recognizing visitors finger tips on the table surface. The other computer will run the EM spectrum application, which we are developing in Adobe Flash. We’ll post more about this exhibit as we continue to make progress.

As we state in the posting… “We seek a full-time exhibit developer who can help us build and install computer-based multi-touch tables and custom exhibit installations. We also need help with technical documentation, email and phone support, so strong writing and communication skills are required. This is an entry-level position, so we don’t expect applicants to have museum or exhibit experience, but we are looking for someone with fabrication, shop skills, electric skills, and some experience with computers. We’re a small company, so we need someone who is flexible and has a diverse skill set.”

Please respond through Craigslist. No Phone calls, please.

Updated Friday May 22, 2009: We’ve removed this listing, thanks to everyone who has applied. We will let you know.

We will be closing our 2009 survey on computer-based interactive exhibits in museums on Monday. If you haven’t done so already,  we encourage you to take the survey.  (Update: May 26, 2009 The survey is closed.) Open Exhibits is a planned open source initiative for informal science education.  The software will be developed in Adobe Flash and Flex, it will include support for multitouch gestures. You can learn more about the project on the Open Exhibits website. Also, we encourage you to join our Facebook group.  

We have received over 120 responses to the survey so far.  We will share the final results, first with those who have taken the survey. Later, we will post all the results on the Open Exhibits website.  ]]> http://www.ideum.com/blog/2009/05/21/open-exhibits-computer-based-exhibits-survey-closing-soon/feed/ 0 http://www.ideum.com/blog/2009/05/06/multitouch-actionscript-developer/ http://www.ideum.com/blog/2009/05/06/multitouch-actionscript-developer/#comments Wed, 06 May 2009 17:48:35 +0000 Jim Spadaccini http://www.ideum.com/blog/?p=841 We’ve just posted a position for an ActionScript developer to work with us in creating next-generation multitouch, multiuser exhibits. The posting is on Craigslist. You can see the full job description here. Instructions for applying are included (please no phone calls). Update May 20, 2009: We’ve removed this job from Craigslist.

Tomorrow, Chris Gerber and I are off to Munich for Natural User Interface’s Multi-Touch Summit 2009. The meeting at the Adobe offices in Germany will focus developing multi-touch applications with Adobe Flash, Flex, and Air. We will be presenting in the afternoon and we’ll discuss our experience in using ActionScript 3 to create interactive exhibits. We’ll also be announcing some very important news (more on that later). A schedule for the summit is available on the NUI Website.

There’s more on the summit on the Natural User Interface (NUI)’s  Multi-touch Blog.

In addition, I am the chair for a session entitled, Open Source for Museums: The Next Experiment in Museum Technology. We have a great panel and one that represents most of the major open source initiatives in the museum world. The presenters are: Scott Sayre from Pachyderm, Tom Scheinfeldt  from Omeka, Carl Goodman from CollectionSpace, and Bryan Kennedy from Science Buzz  who will explain how Drupal was used in development of their site. In addition to these direct connections, some in the group (myself included) are advisors on other open source projects such as Fluid Engage and Steve. 

Last year, I wrote an article for the National Association for Museum Exhibition’s Journal, Exhibitionist, where I interviewed most of the panel members about their projects.  The article was reposted in the Ideum blog see Open Source Software: New Possibilities for Museums. The article has also just recently been translated into Czech and appears in Muzeum, the magazine of the Narodni Muzeum in Prague.  

In the AAM session, I won’t be talking much about Open Exhibits, as we are going to focus on projects that are already fully underway. (We are resubmitting our proposal to NSF and we are are still looking for museum professionals to fill out our survey on computer interactives in museums.)  I look forward to seeing some of you at AAM in Philadelphia.

Update May 5, 2009: I will be posting PDFs of the slides presented during the session. Here’s the introduction to Open Source Software: New Possibilities for Museums (PDF 900K). It includes links to all of the projects mentioned and references.

Update May 6, 2009: Two more presentations in PDF form here. Omeka: Open Source Web Publishing for Museums (PDF 2Megs) by Tom Scheinfeldt and Museum Open Source: Make or Break – Pachyderm Retrospective (PDF 740K) by Scott Sayre.

Update April 20, 2009: Shelly Mannion has some photos of multitouch table at MW2009.  You can see the photos that are tagged, multitouch in her Flickr group.

Also, here is a picture we took of our multitouch table and the Microsoft Surface. We were moving out the our exhibit space, when we came across a Microsoft Surface set up for the a demo the next day. You can see more pictures in our Flickr set multitouch exhibits.

Update April 17, 2009:  Paula Bray wrote a story about the workshop for Fresh + New(er) blog entitled, MW2009 – Multi-touch: what does this technology hold for future museum exhibits?

Update April 16, 2009: Thanks to everyone who attended it was a fun day. Here’s the Introductory Presentation and the Design Considerations presentation in PDF format.

We received a New Mexico Technology Solution Award for the MT2 Multitouch, Multiuser Table.  Some of the other awardees included Intel Corporation, Nuvita, Qwest, and Cisco. You can learn more about the New Mexico Technology Council and the TechEx Awards on their Website.

There’s more about the project and the results from our 2008 survey on OpenExhibits.org. Last year we received 125 responses to the survey, we are trying to match that number this year. Also, we encourage you to join our Facebook group. We’ll be posting the results of the survey in late spring on the Open Exhibits website.

For our second multitouch table installation, we designed a mapping and photo mashup application. (Similar to the one we completed for our first table installation at the Don Harrington Discovery Center in Amarillo, Texas)  For Vulcan Park and Museum, we developed a multitouch panoramic viewing application that runs on two HP TouchSmart kiosks.

Like the touch table exhibit, these kiosks use NUI Snowflake software with custom Flash software which we developed.  The panoramic viewer allows for simple pinch gesture to zooming of a panoramic photograph that we took from a top the Vulcan Park statue.  Users can also flip (or flick) photographs in a photo viewer window that is connected to points of interest on the panoramic image. We designed and developed the stand and exhibit-case for the HP TouchSmart.  This platform provides a low-cost touch and  multiouch platform for computer exhibits.  There’s more about the panoramic viewer application in our portfolio.

Here are some photographs for the opening party for Birmingham Today.

Visitors interact with the interactive map and photographs of Birmingham.

You couldn’t tear some visitors away from the touch table.

The multitouch-enabled panoramic viewers were placed by the large windows in the gallery space allows visitors to explore the Birmingham Skyline.

The first post addresses user interaction and feedback, the second focuses on User Interface (UI) elements, objects and environments, while the third looks more broadly at how multitouch and multiuser exhibits can shape the visitor experience.

Traditional Computer-Based Interactives
Part of our attraction to multitouch and multiuser exhibits has to do with their ability to enhance the visitor-experience. Much of the criticism surrounding traditional computer-based exhibits and kiosks is valid:

1.    They tend to isolate visitors
2.    They are often too information heavy
3.    Interactivity is often limited
4.    In many cases the experience can be easily replicated at home, school or work

The traditional computer-based exhibit seems to have more in common with an ATM than it does with other interactive exhibits found in the museum. Traditional computer-based interactives can be functional, but are not often inspiring.

Multitouch, Multiuser
Making computer-based exhibits more interactive and engaging has always been the challenge. Ideally, these exhibits will act as catalysts for visitor communication and conversation. In the past we’ve developed computer-based interactives that have used push-button interfaces (also see Jukebox Memories), touch capacitors, and even spherical displays.  Multitouch and multiuser capabilities are a very welcome development, and I don’t think I’m overstating the case by saying it may be the most important innovation for computer-based exhibits since the development of the World Wide Web.

Multitouch, multiuser exhibits present a number of advantages over their single-touch, single-user counterparts:

1.    Multiuser exhibits bring museum visitors together and encourage social interaction
2.    Interaction is more physical and intuitive
3.    Multitouch tables are still novel and present an experience not found at home, school, or work

With the popularity of large-screen TVs and gaming platforms like the Wii, we weren’t really sure what type of response we would receive during our initial testing. But we were really overwhelmed with the enthusiasm for the platform. There is something very compelling about the table top format itself. It allows visitors not only to interact with the program running on the surface, but also with each other.

Even with early prototypes, we found the table drew large crowds and kept visitors’ interest.  How long this novelty factor lasts is hard to judge. Obviously, good design with interesting content is required to make great exhibits. Looking forward, multitouch and multiuser capabilities have a lot to offer the future of exhibit design.

Hardware, Now More Software
Since last summer, we’ve had to work on both hardware and software solutions. However, now that we have a stable multitouch hardware platform, we are shifting focus onto software design and development along with more in-depth visitor testing. I’ll keep posting what we find. Your thoughts and comments are welcome.

Finally, if you’re looking for more about multitouch you can visit the Multitouch Blogs directory site or the following sites:  Interactive Multimedia Technology, The NUI Group Blog , Natural User Interface blog,  Point & Do, Microsoft Surface Blog,  MultiTouch Blog, Multitouch Barcelona,  Stimulant, and Touch User Interface.

The first post addresses user interaction and feedback, the second focuses on User Interface (UI) elements, objects and environments, while the third looks more broadly at how multitouch and multiuser exhibits can shape the visitor experience.

Objects, Environments, and Navigation
The last post explored the types of gestures and the way in which visitors interact. This post focuses on the elements that visitors interact with. As you’ll see, there is strong focus on objects.  Multitouch and multiuser applications also require that we rethink standard navigation schemes.

Objects. Resizing and rotating objects is a common feature in many multitouch applications. This type of interaction is very intuitive and we observed museum visitors spending a lot of time viewing photographs in this way.  For a multitouch multiuser table, an object-based approach makes sense. Objects can be rotated to accommodate visitors from any side of the table and multiple users can interact independently with objects.  The extension of this object-based approach to connect with museum collections seems natural.

Environments. There are a number of interesting possibilities when we look at environments in multitouch design. In the Yahoo! Maps and Flickr mashup application we developed, the map was the environment. It provided context for the geocoded photographs. However, the environment could be even more dynamic with ability to transform objects or trigger new actions.

For example, one could imagine an exhibit where visitors could move objects from one-side of the table to other—this movement could represent time and we could see that object or photograph change over time. The table could effectively work as a dynamic timeline: move right and move forward in time, move left and you move back.

Additionally, a table environment could be divided into zones, and as objects are moved into these areas events are triggered, or objects are transformed in some way.  There are interesting possibilities here for works of art. Exhibits where visitors manipulate sketches which transform into finished paintings could be developed.   For scientific images, different views of objects shot by different types of imaging equipment or in different wavelengths could be used. Finally, developing physics games where gravity, the coefficient of friction, or other parameters could be changed is another possibility for science exhibits.

Universal, Omni-Directional Navigation. Since multiuser, multitouch applications can be accessed from any side of the table, there is a need, in some applications, for a universal navigation element.  In our mapping application, we wanted to provide museum visitors with the ability to control the map environment. We found that panning could be a shared function, but the ability to zoom and select the map-type  couldn’t be shared in the same way.

Visitors can “call the orb,” with a quick double-tap on the table surface and the orb even contains a compass to help the visitors orient themselves to the map they are controlling.

The Microsoft Surface team recently released an example of Newsreader with universal, omni-directional navigation.  In this case, the navigational element is fixed and rotates around the center of the table.

Symmetric Navigation. Another possible approach to navigation on a large table where users can approach from any direction is what we call symmetric navigation. In this case, the navigational items are repeated in two places on the table. For example, a simple menu could be present in opposite corners of the table. Another configuration would have sets of tabs or buttons appearing on opposite sides of the table.  The information would be repeated, but would be properly oriented for each side.

User Interface Elements
Many of the user-interface elements are similar to those found in touch screen mouse-driven applications. However, some new elements have appeared, mostly attributed to the development of the iPhone.

Buttons /Icons/ Thumbnails. These conventional navigational elements appear in most traditional applications. Of course, these graphic elements need to be large enough to pressed by a finger tip, like they would in a touch screen application.

Dials. Apple’s iPhone uses “slot machine-type” dials for some of the applications.  While I haven’t seen these dials appear in other multitouch applications, one could imagine these finding their way into computer-based exhibits.

We explored the possibility of using dials, although ours were right-to-left oriented. This was a very early concept for a floor-to-ceiling multitouch enabled health exhibit.

In this rough mock up we imagined that users would be able move the dials right to left and see changes in life expectancy and other important health information.

Drawers. Pull-down menus don’t work well for touch, or multitouch applications. Although, a variation requiring two touches can be used: one touch opens the menu (or drawer) and the second touch makes a selection. This can be used with symmetric navigation or for universal, omni-directional navigation.  Additionally, drawers can be used to provide additional information, not just navigational choices.

Flip (or Flick) elements. The HP TouchSmart and Apple iPhone have applications that allow visitors to “flip or flick” through photographs.  We used a similar photo viewer for our panoramic viewer exhibit.

In this exhibit, visitors can pan and zoom an image of the Birmingham skyline and click on points of interest. Each point has a photograph a visitor can select from the panoramic image or flip (or flick) through using the movable photo viewer. I should mention this exhibit is developed for the HP TouchSmart.  It is part of an installation for Vulcan Park and Museum in Birmingham, Alabama.

As you can see, there are some challenges and interesting possibilities when it comes to navigation and design for multitouch exhibits and applications.   In my next post, I’m going to look more broadly at how multitouch and multiuser exhibits can shape the visitor experience.

We developed these applications for our multitouch table (MT²) and for the HP TouchSmart platform. After developing touch screen exhibits for nearly a decade, the differences between standard touch and multitouch are very much in focus.

From the beginning, it has been clear that mouse or even standard touch-screen conventions wouldn’t be completely applicable. Multitouch and multiuser design requires new thinking, more experimentation, and careful user-study.  I want to share some of what we’ve learned and the areas that we are still investigating.  I’m also doing this in preparation for a workshop that we’ll be conducting at Museums and the Web (called “Make it Multitouch”) and a short presentation for the Canadian Museum Association’s annual meeting (called “Doers and Dreamers“) in Toronto at the end of March.

This discussion has been divided into three blog posts: The first explores user interaction and feedback, the second focuses on User Interface (UI) elements, objects and environments, while the third looks more broadly at how multitouch and multiuser exhibits can shape the visitor experience.

Interactions
How do users interact with interface elements and content on a multitouch screen or surface? And how are these interactions different than those we observe in standard mouse-driven or touch-screen applications? Below is a list of some of the unique ways visitors can interact with a multitouch interface. As you’ll see, some are very natural and others are more obscure. It is a strange blend of intuitive gestures and secret handshakes.

Touch. The same as standard touch screen interactions, touch areas are made larger to accommodate a finger tip  than those for mouse or trackball driven kiosks and exhibits.

Drag. With either one finger or multiple points, this type of interaction is similar to what we see with a mouse and pointer.

Pinch & Expand. This is an intuitive way to increase or decrease the size of objects in multitouch environments. In one case, we saw that just the act of placing a hand on the table surface slightly expanded an object (the hand opened a bit more as it impacted the surface). This allowed the visitor to immediately understand how to size the object. Pinch & Expand is common in ordinary hand gestures when talking about how big or small something is.

Rotate. As a visitor drags or pinches and expands an object it becomes apparent whether it can be rotated or not. Since multitouch tables have multiple points of approach, most applications provide visitors with the ability to rotate objects.

Double-Tap. We’ve used this type of interaction in a mapping mashup to “call over” a floating universal navigation element. We found this helpful for our large table, where the floating navigational item could be out of reach. However, our testing showed that this was not as an intuitive as some of the other types interaction. Although, once observed, most visitors found it simple and helpful.

Draw. Some multitouch applications allow visitors to draw shapes, such a NUI Gravitor application (seen above). It is also possible to draw “commands.” For example, you could draw an “x” on an object to close it. This would assume, however, that the object could not be dragged or resized, since those interactions would be interfere with the ability to draw.

Flip or Flick. It allows visitors to quickly browse through “stacks” of photographs or other fixed size objects. This works well with “dual touch” technologies like the iPhone and the HP TouchSmart.

Feedback
Visitors can benefit from additional feedback as they interact with multitouch applications. Occasionally, there can be a lag in direct feedback for some of the interactions listed earlier. This can be especially true in multiuser environments where the application is trying to process dozens of simultaneous points.

Tracers/Trails/Auras. As each finger point is detected as a “blob” by the “touch core” software, a small graphic or animation follows the point across the surface or screen. You can see a tracer (above) following the visitor’s finger as he resizes the painting. (His finger is slightly off the table so you can clearly see the “tracer.”)

Highlights and Ghosting. As visitors touch an object, it can be made to highlight or animate in some way. Ghosting can be helpful for dragging as you can still see where the item originated. Highlights provide the user with instantaneous feedback and reinforcement of their current action.

Connections. Lines (or other indicators) that connect objects can be helpful in way-finding particularly in multiuser environments. For our multitouch mapping application, we created connection lines from photographs to their points on the map—knowing that one user may be manipulating a photograph while another is controlling the map. This allows a visitor to trace the connection line back to the geographical point where the photograph was taken.

In my next post, I’ll explore how these interactions are applied to User Interface (UI) elements, objects and environments. As you’ll see, things get really interesting when we look to adapt and invent new ways for visitors to interact.

The table runs a custom mulituser, multitouch application we developed with the Don Harrington Discovery Center and Vulcan Park and Museum. This multitouch mashup application uses Flickr and Yahoo! Maps. There is more on the design and software development process on the Ideum portfolio. The video below shows some of the features found in the application.

The press came out to see the exhibit. The local newspaper and all three network news channels showed up. Below DHDC’s Executive Director, Joe Hastings got interviewed by the local press.

Our next installation is in two weeks in Birmingham, Alabama. We’re going be installing another table along with two multitouch enabled HP TouchSmart kiosks. As far as we know, this will be the first time multitouch technology has been used exclusively throughout a permanent exhibit space.

This mashup site is open to visitor-contributions and the photographs are drawn from a Flickr Group which is also called, New Mexico Stories.  The group is administered by the Foundation. I blogged about the planning process last October (see Planning for Social Sites).

New Mexico Stories  pulls geocoded images from Flickr and places them on a map of New Mexico. Images are divided by county, so the site should be able to easily hold thousands of images and remain, “navigable.” In addition, a Gallery view creates a random collage of 50 images from the collection.

The site has just become available and the Foundation will be formally announcing its release later this spring. Please let us know what you think, and if you have a New Mexico Story to share, please do.

The idea of the directory is to raise awareness and increase the authority of sites focusing on multi-touch design and technology. Multitouch Blogs creates a combined RSS feed and there is a Web widget with links to individual posts and blogs.  There is also an OPML file with links to each of the sites in the directory. Many blogging platforms support OPML and it can be used to create a quick blog roll.  We hope to see the directory grow over the next few months.

For our first multi-touch application we are creating a mashup using Yahoo! Maps and Flickr. This mashup is being developed with the Don Harrington Discovery Center and Vulcan Park and Museum (located in Birmingham, Alabama).  They’ve been great partners as we’ve worked through the conceptual and design challenges that a multitouch, multiuser application presents. You can learn more about this multitouch mashup project on our portfolio. We’ll be installing the tables in March.

We’ll be showing the table and the mashup application in April at Museums and the Web 2009 in Indianapolis.  We will have an exhibit booth and we will be holding a full-day workshop called, “Make It Multitouch.” We hope to see you there.

EditorOne now includes the display of source clip information and metadata. For example, when a visitor create a video mashup, a page is created that includes descriptions, transcripts, attributions, copyright information, and other important collections information. (You can view a live example here.)

EditorOne has new hosting and installation options too.  You can learn more about EditorOne’s features and you can try the demo version.

Museum Blogs and Museum Podcasts have integrated widgets for viewing all posts and episodes in the directory. In addition, there is a ”detail” page for each blog and podcast each with its’ own individual widgets.  All of these widgets can be freely cut-and-pasted into other Websites or blogs. Both directories accept new blogs and podcasts, so if you have any additions please send them along.

In my course we’ll be spending time within the online teaching environment and taking “field trips” to some of the social networking sites that museums are beginning to connect with. The TEC-CH online program has a full compliment of credit courses (European Credits) taught by professionals from all over the world. You can see the full course list here.