How to Remove Duplicates from a Python List

Removing duplicate elements from a Python list is a common task in data processing and manipulation. Here are several effective methods to achieve this:

Methods to Remove Duplicates from a Python List:

• Using set():
• Using OrderedDict():
• Using List Comprehension with not in:
• Using a Loop and a Temporary Set:

1.Using set():

• Principle: Sets in Python inherently store only unique elements. This method leverages this property for efficient duplicate removal.
• Implementation:

def remove_duplicates_set(lst):
    return list(set(lst))
my_list = [1, 2, 2, 3, 3, 4, 4, 4, 5, 1,5]
unique_list = remove_duplicates_set(my_list)
print(unique_list)

Output:

[1, 2, 3, 4, 5]

2. Using OrderedDict() (Preserving Order):

• Principle: If you need to maintain the original order of the elements in the list, you can use an OrderedDict from the collections module.
• Implementation:

from collections import OrderedDict
def remove_duplicates_order(lst):
    return list(OrderedDict.fromkeys(lst))
my_list = [1, 2, 2, 3, 4, 4, 5, 1]
unique_list = remove_duplicates_order(my_list)
print(unique_list)

Output:

[1, 2, 3, 4, 5]

3. Using List Comprehension with not in:

• Principle: This method iterates through the list and creates a new list containing only elements that have not been encountered before.
• Implementation:

def remove_duplicates_comprehension(lst):
    unique_list = []
    for item in lst:
        if item not in unique_list:
            unique_list.append(item)
    return unique_list
my_list = [1, 2, 2, 3, 4, 4, 5, 5, 1,6]
unique_list = remove_duplicates_comprehension(my_list)
print(unique_list)

Output:

[1, 2, 3, 4, 5, 6]

4. Using a Loop and a Temporary Set:

• Principle: This approach iterates through the list and adds elements to a set. Since sets only store unique values, it effectively removes duplicates.
• Implementation:

def remove_duplicates_loop_set(lst):   
    unique_set = set()
    unique_list = []
    for item in lst:
        if item not in unique_set:
            unique_set.add(item)
            unique_list.append(item)
    return unique_list
my_list = [1, 1, 1, 2, 2, 3, 4, 4, 5, 1, 7]
unique_list = remove_duplicates_loop_set(my_list)
print(unique_list)

Output:

[1, 2, 3, 4, 5, 7]