Skip to content

fix: handle empty tokens

Michelle Weidling requested to merge fix-empty-tokens-for-collation into develop

This MR ignores empty token in the JSON creation.

During the normalization of the texts, some code points for diactrics and other special characters are removed in order to simplify the collation process. As a result, normalized tokens are sometimes empty strings which cause CollateX to throw an exception.

Merge request reports