Tokens and Character Set in Python

In this tutorial, we’ll explore tokens and character sets in Python, and understand their significance in code comprehension and manipulation.

Tokens:

In Python , tokens are the smallest individual units of a program. Tokens are building blocks of python code they help us to construct commands in the program.There are different types of tokens which include:

Keywords:

Keywords are reserved words that have special meaning in Python and cannot be used as identifiers and possess unique qualities.They are 33 keywords in Python. Examples include if, else, for, while , def , class , import, and return.

Example:

x = 10
if x > 5:
    print("x is greater than 5")
else:
    print("x is not greater than 5")

In the above example, if and else are the keywords.

Identifiers:

Identifiers are names given to variables, functions, classes, modules, or other objects for their identification. While naimg identifiers there are some important rules to be followed as python is a case sensitive language.Few rules include such as starting with a letter or underscore, and can contain letters, digits, and underscores.

Example:

variable_name = 10
def my_function():
    pass

In the above example variable_name and my_function are identifiers.

Literals:

Literals represent fixed values in Python code. There are several types of literals like:

• Numeric literals: Integer (10), Floating-point ( -0.001), and Complex (2+3j).
• String literals: Enclosed in single (‘hello’) or double (“world”) quotes.
• Boolean literals: True and False.

Example:

integer_literal = 42
float_literal = 3.14
string_literal = "Hello, world!"
boolean_literal = True
none_literal = None

Here 42, 3.14, “Hello, World!”, True, None are literals.

Operators:

Operators perform operations on operands and produce a result. Python supports various types of operators, including arithmetic operators (+, -, *, /), comparison operators (<, >, ==, !=), logical operators (and, or, not), assignment operators (=, +=, -=), and more.

Example:

result = 5 + 3
comparison_result = (result > 5)
logical_result = (result > 5) and (result < 10)
assignment_result = result * 2

In the above example ‘+’,’>’, ‘*’ are operators.

 

Character Set:

In Python, a character set refers to the collection of characters that can be used to write code and represent textual data. Python supports a diverse character set that includes alphanumeric characters, punctuation symbols, whitespace characters, and special characters from various languages and scripts.

Example:

# Alphanumeric characters
variable_name = "hello123"

# Punctuation symbols
punctuation = "!@#$%^&*()-_=+"

# Whitespace characters
whitespace = " \t\n"

# Special characters from different languages and scripts
special_characters = "你好, नमस्ते"

# Combined example
combined_example = "Hello, 你好, नमस्ते! 😀"

Here variable_name contains alphanumeric characters, whitespace contains whitespace characters(‘ ‘,’\t’,’\n’).