Pentelligence: Handwriting Recognition with Motion and Audio Data from Pens

Pentelligence is a novel interaction device that does not require special paper or additional hardware (except for a computer) to recognize handwriting. This pen can achieve recognition rates of over 98% for digits on individual users without writing instructions. Its main strength lies in the use of deep neural networks that learn the individual writing styles of the users. By combining motion and audio data from inside the pen, the strengths of the individual sensors complement each other.
On this page we offer a short introduction to our project and give a brief overview of the main features of Pentelligence. For further information, please refer to our paper: Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition

Developed by the Human-Computer Interaction group at the University of Hannover.

Click to watch video

Vision

Note-taking in the digital age?

Click here to learn more about the vision of Pentelligence.

Hardware

Take a look inside and explore the concept!

Click here to see how the heart of our digital pen is designed.

Recognition

Audio & motion data from pens?

Learn more about recognizing handwritten digits.

Note-taking in the digital age

For thousands of years humans wrote down their ideas in various ways. Even today, the pen is an indispensable part of our lives. Although even tablets offer precise input options, many people still prefer the pen and paper.

The vision of Pentelligence is to bridge the analog and digital worlds with a device that doesn't differ from the appearance of ordinary pens. In our paper at the CHI 2018 we have shown the concept and the recognition of digits, which is particularly important because, in contrast to words, in many cases no subsequent autocorrection can be performed.

The project will continue to pursue the vision of a digital pen that does not require special paper, any other additional hardware or even cameras to recognize handwriting.

A look inside

From the inside, Pentelligence relies on small, inexpensive and easily available hardware components. With a length of 12cm and a diameter of 12mm, the prototype almost reaches the dimensions of a conventional ballpoint pen in the first phase.

A microcontroller with an easy-to-program Arduino bootloader records the measured values of the microphone and the inertial measuring unit together with a specially self-developed binary writing pressure sensor and sends them to the computer via USB.

In future releases a wireless version will be developed, which will allow an even more realistic look and feel compared to ordinary pens.

Deep learning

Handwritten digits are as unique as their writers. For this reason, modern methods of deep learning are necessary to customize Pentelligence to the individual writing style.

The envelopes of the writing audios, fed into the neural networks, eliminate surface-related features. Downsampled and averaged motion data together with write pressure information capture individual strokes and reconstruct the writing trajectories within the classifiers.

By subsequently combining the results of the audio and motion classifiers, the complementary features can be merged and recognition rates of over 98% can be achieved.

2018
Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition Maximilian Schrapel, Max-Ludwig Stadler, Michael Rohs Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
Full Paper
        

Pentelligence is developed by the Human-Computer Interaction Group at the University of Hannover. For additional details, read our paper (available on our homepage or in the ACM digital library). For inquiries, please contact: maximilian.schrapel@hci.uni-hannover.de

© Copyright Human-Computer Interaction Group, University of Hannover 2018