[core][fr] lexicographer: fix crash when empty list of tokens