Coders Packet

Tokenizing String in Java using Built-in Function

By Priyonath Mondal

  • Tokenizer.java
  • Tokenizer.class
  • The String Tokenizer class allows an application to break a string into tokens. A token is returned by taking a substring of the string that was used to create the StringTokenizer object.

    String Tokenizer is a class that breaks a string into multiple tokens. Let us consider a string as, “JAVA PYTHON SQL AI”.

    Now from the above string assume we have to parse wherever there are spaces to create smaller strings. That is, these are the smaller strings that has to be created:

    JAVA

    PYTHON

    SQL

    AI


    All these smaller strings which are created after parsing a string are only called as “Tokens”.

    We will make use of a built-in class called "StringTokenizer".

    Things to be known about StringTokenizer:
    Delimiters are those characters which separates the tokens.

    Object of StringTokenizer can be created in three different ways:
    • StringTokenizer st = new StringTokenizer( String str );
    str is the string to be tokenized. This case, considers the default delimiter as space.
    • StringTokenizer st = new StringTokenizer( String str, String delim );
    delimeters is set of delimiters whcih are used to tokenize a string.
    • StringTokenizer st = new StringTokenizer(String str, String delim, boolean flag);

    Download Complete Code

    Comments

    No comments yet