tokens_merge() merges two tokens objects x and y into a larger
tokens object. tokens_merge_all() merge all the arguments into one
tokens object. The result is a concatenation of the tokens, in which the
order of the items in the input is preserved.
Value
An object of class tokens.
Examples
(tks1 <- tokenize(c("This is a first sentence.")))
#> Token sequence of length 5
#> idx token
#> --- --------
#> 1 this
#> 2 is
#> 3 a
#> 4 first
#> 5 sentence
(tks2 <- tokenize(c("It is followed by a second one.")))
#> Token sequence of length 7
#> idx token
#> --- --------
#> 1 it
#> 2 is
#> 3 followed
#> 4 by
#> 5 a
#> 6 second
#> 7 one
(tks3 <- tokenize(c("Then a third one follows.")))
#> Token sequence of length 5
#> idx token
#> --- -------
#> 1 then
#> 2 a
#> 3 third
#> 4 one
#> 5 follows
tokens_merge(tks1, tks2)
#> Token sequence of length 12
#> idx token
#> --- --------
#> 1 this
#> 2 is
#> 3 a
#> 4 first
#> 5 sentence
#> 6 it
#> 7 is
#> 8 followed
#> 9 by
#> 10 a
#> 11 second
#> 12 one
tokens_merge_all(tks1, tks2, tks3)
#> Token sequence of length 17
#> idx token
#> --- --------
#> 1 this
#> 2 is
#> 3 a
#> 4 first
#> 5 sentence
#> 6 it
#> 7 is
#> 8 followed
#> 9 by
#> 10 a
#> 11 second
#> 12 one
#> 13 then
#> 14 a
#> 15 third
#> 16 one
#> 17 follows
tokens_merge_all(list(tks1, tks2, tks3))
#> Token sequence of length 17
#> idx token
#> --- --------
#> 1 this
#> 2 is
#> 3 a
#> 4 first
#> 5 sentence
#> 6 it
#> 7 is
#> 8 followed
#> 9 by
#> 10 a
#> 11 second
#> 12 one
#> 13 then
#> 14 a
#> 15 third
#> 16 one
#> 17 follows