Sekai World releases its Live2D plugin

Posted on Edited on In Views:

TL;DR

Recently, Sekai Viewer released the Live2D Viewer function. I forked a plugin package called find-live2d3, did some modification and renamed to @sekai-world/find-live2d-v3. Release of my package can be found on npmjs and GitHub.

What is Live2D?

Live2D is a software technology that allows you to create dynamic expressions that breathe life into an original 2D illustration.
Live2D is a technique of generating animated 2D graphics, usually anime-style characters, using layered, continuous parts based on a single illustration, without the need of animating frame-by-frame or creating a 3D model.
What is Live2D? | Live2D Cubism

Live2D in Project Sekai

Project Sekai use Cubism 3.3 to make the Live2D models. As Cubism promised, all Cubism Live2D SDK version 3+ has backward compatibility. Although the only one downloadable SDK is version 4.0, it works well with models from 3.3.

But not like other unity games which also use Live2D, Colorful Palette made some modification on that. To extract the Live2D models from Project Sekai asset bundles, I tried Perfare’s UnityLive2DExtractor at first, unfortunately it doesn’t work. Then I realized a painful fact: I have to extract motions and expression from the bundle files by myself.

Then I dived into the code of unity live2d extractor to figure out how it works. I believe those code is helpful cause it works with many other games. After that I made it clear how to reconstruct motion json files, I wrote some python code and amazingly they worked! But one big problem is still blocking my way: the field Parameter is a number, not a string, and I don’t know what it means. In other games, Live2D motion bundle has a unity GameObject component in it, it doesn’t appear in the files of Project Sekai. Colorful Palette either hide the GameObject somewhere in other asset bundles, or use another way to determine the Parameter field.

After a lot of reasearch (which costs me about two days), I found some clue from Cubism Live2D Unity examples. From Perfare’s code I knew that the Parameter number is crc32 computed from some strings (usually from GameObject), but I’m not sure how was the strings composed. I did some tests and found that I just missed some prefix of the strings. However, I still didn’t know which parameters are available for those motion files.

I took a look at Cubism Live2D Web SDK, it contains a core module which can decode the moc3 files and read out parameter list. This module is compiled from c/c++ code through EMscripten into js code. I’m not here to decode the machine code, so with my last hope I opened the moc3 file in hex editor. I felt lucky because the structure of parameters in moc3 file is pretty simple. In the start of moc3 file it has some chunks like offset table, telling the offset of each part. The position of parameter table in the offset table is constant, therefore I can read same position to get the offset of parameter table. The parameter table consists of a lot of strings indicating the name of parameter. Tada, I’m on the right way.

I add some simple code to my python module and most of the parameters are recovered. After some smoke test of my extracted motion jsons I can finally step further.

Implement Live2D in React

I have to say, Cubism has a very shitty documentation support for the WebGL SDK. Luckily they wrote a good production-ready working example of the SDK, but it still doesn’t meet my need. After searching on npmjs, I found a package called find-live2d3. It’s really amazing, transformed the official example into a plugin package which can be used out-of-box. I use this package as a good start.

This package did not provide a GitHub repo address, so I installed it locally and copied everything from node_modules folder. I have to modify this package because:

  • The original package did not allow me to specify a canvas container, it creates a full screen container by default.
  • The model jsons of Project Sekai does not contain motions and expressions, I have to load them in another way.
  • It uses outdated core and framework, which breaks the display of Project Sekai models. I have to update and patch them again.

Therefore I added el argument to initialize function to allow me give my custom canvas container element. I also added a loadMotions function to model instance which allows me to load motions after loading the model.

To help others who may also need a plugin package of Live2D, I make it open source. The documentation is not done, I suggest you to read the code.

Conclusion

As my other datamining work, I learned some useless knowledge again. I figured out how to read moc3 file, how to reconstruct the motion json from AnimateClip of unity Live2D bundle and how to decode the Parameter field of unity Live2D bundle. Furthermore, during the implementation of Live2D Viewer, I knew how to resize a canvas upon window resizing, how to save image in WebGL context. And I do test two models showing at once, it would help the development of Live2D Story Reader.