(More) About Kanji Storyteller

Compose View

Explore View

I've been developing an educational sketch recognition based program called Kanji Storyteller that acts as a learning space for students of foreign languages. Currently, it only supports Japanese and only recognizes a small set of kanji. When I first started working on it for a Computers and New Media course project, I was working with a group of students from Dr. Hammond's Sketch Recognition Lab and I was pretty dependent on their contributions for the recognizer. I want to improve the recognizer and free myself from dependence on others for future development.

Recognizer Update

We ended up using a template based recognizer that checks sketches against a training set on the fly. It can recognize the symbols but it gets too slow with increasing size of the training set and it tends to be a little too temperamental and unpredictable. As such, I will be converting the recognizer to a gesture based recognition scheme.

• Gesture/Stroke Based Recognition

Recognizer Expansion

For Japanese, I'd like to expand the recognizer to recognize:

• 平仮名
• hiragana - phonetic syllabary
• 片仮名
• katakana - phonetic syllabary for words borrowed form other languages
• ローマ字
• romaji - Roman alphabet characters
• 部首
• bushu - "radicals" - Parts of Kanji
• Stroke Order and Direction Based Recognition - rejects or notifies when a kanji written in the wrong order or direction
• More Kanji and Jukugo

Social Connection and Collaborative Learning

One of the major features that I wanted to include was support for networked connections where other students, mentors, instructors, and even spectators could join a session (class, group) and share their ideas and thoughts about the kanji. The shared interaction space would allow the students and other participants to share mnemonics, demonstrate correct stroke order, and collaboratively create scenes that would help reinforce learning. Some classes of participants could be restricted to only observe and comment while other classes of participants could add to the scene annotation and sketch as well as comment. Providing audio support for open dialogue would be extremely beneficial.

• Host a session (teacher only?)
• (Sockets?) connect to a session
• Support Audio Chat
• Group editing of scene annotation
• Canvas updates across multiple connected sessions in real time
• Networked saving and storage of canvas files (xml)
• Suggest beneficial group behavior
• mnemonics
• memorable stories about kanji
• discuss aspects of kanji
• make fun/cool stories in the annotations
• group games (competitive and collaborative)

Gamified Learning Suite

Going beyond character recognition, I would like the application to provide a language learning suite that featured grammar lessons, full sentences, vocabulary drills, and learning games and quizzes. Kanji Storyteller should also keep track of the student's progress in the form of the kanji and concepts that they have encountered and mastered. Making this generic and modular enough to add new languages would be ideal, although there are Japanese specific concepts that would need to be emphasized. (Better to do one thing well than many things poorly.)

• Grammar Lessons
• Example Sentences
• Vocabulary Drills, Learning Games, Quizzes
• Progress Tracking
• Kanji Encountered
• Kanji "Learned"
• Grammar Concepts Mastered
• Badges / Achievements
• Learning Metrics
• Average Difficulty of Kanji Encountered (Breadth of Knowledge)
• Grade Level
• JLPT
• Average Difficulty of Kanji Mastered
• Average Time to Recall and Write Kanji
• Average Time to Complete Drill of N Kanji
• % recall of Kanji characteristics (Depth of Knowledge)
• Onyomi, Kunyomi, English Meaning, Jukugo
• User Profiles Stored on Database
• User Profile Image

Dynamic Sticker Image Lookup and Related Kanji Image Creation

To provide some variety in the kind of images that can be used for the stickers, I'd like to move away from using a static image library/database and lookup images on the internet dynamically at runtime. This would also reduce the amount of work it takes to add new kanji to the program where only the training set, classifier, and audio would have to be updated. The blue kanji identifier tag on the bottom right of the image would have to be added on the fly too though.

• Dynamic Sticker Image Lookup
• Dynamically Create Blue Kanji Identifier Tag
• Dynamically Create Basic Kanji Image
• Dynamically Create Basic Jukugo Image

Print Function 

Students and Teachers may want to have a paper printouts of the information for their own records or creations. 

• Dictionary Panel
• Canvas
• Comments Panel (or subset of it)
• Story Panel (or subset of it)

Miscellaneous

• Change the layer control button icons to images rather than letters
• Get rid of the 'flashing' effect when things are moved/updated in GUI
• Search Function to better search English to Japanese
• Translate Multiple Stickers at once
• Move stickers relative to background if size is changed
• Get rid of 0 value for Frequency of Use column in Database
• Smarter/Better GUI sizing
• Have quick dictionary lookup by using kanji that is already on the canvas

Other Languages
At some point, I may also like to expand the whole application to support other languages (but I'm not completely sold on the idea):

• English
• Dictionary Panel
• Roman Alphabet
• Words
• Korean
• Hangul
• Detecting Syllabic Groupings
• Detecting Symbols within Syllabic Groupings
• Detecting Valid Combinations and Arrangements of Letters Within Groupings
• Chinese (Mandarin)
• Dictionary Panel
• Simplified Kanji Set
• Update and Generalize Menu and Change Menu Options Based on Language

Dictionary Panel - Sortable Search

Dictionary Panel - Kanji Information Card