How To Use Proxies To Rotate IP Addresses In Python? 5 Easy Steps In 2024

When you use a proxy, you do not directly connect to the target server; instead, you direct your request to the proxy server that evaluates it and returns a response. Proxies have many benefits, such as they hide your real IP addresses, bypassing filters, censorship, etc.  It is sometimes difficult to perform advanced web scraping

Table of Contents

• How to Use Proxies to Rotate IP Addresses In Python Import LibrariesCreate A FunctionRead The .txt FileUse a For LoopRun the Script
• Import Libraries
• Create A Function
• Read The .txt File
• Use a For Loop
• Run the Script
• Tips To Build A Proxy Rotator  Use Reliable, Free Proxy ServicesRefrain From Using Predictable Proxy IP AddressesConsider A Premium Proxy ServiceGo For Elite ProxiesCombine IP Rotation With User Agent Rotation
• Use Reliable, Free Proxy Services
• Refrain From Using Predictable Proxy IP Addresses
• Consider A Premium Proxy Service
• Go For Elite Proxies
• Combine IP Rotation With User Agent Rotation
• FAQs:
• Conclusion

When you use a proxy, you do not directly connect to the target server; instead, you direct your request to the proxy server that evaluates it and returns a response. Proxies have many benefits, such as they hide your real IP addresses, bypassing filters, censorship, etc. 

It is sometimes difficult to perform advanced web scraping because of sending multiple requests from the same IP address. Web scrapers face the common problem of getting blocked by websites they scrape. But they can use many techniques to prevent getting blocked, such as

• Rotating IP addresses
• Using headless browsers
• Reducing the crawling rate,
• Using proxies

A proxy rotator can help you get around most of the anti-scraping measures. You can also use it to sidestep any rate limits implemented on the target website and extract data successfully. But what is a rotating proxy? It is a proxy that assigns a new IP address for every connection from the proxy pool. It means you can send 1000 requests to any number of websites by launching a script and getting 1000 different IP addresses.

This article helps you to understand how to use proxies to rotate IP addresses in python.

How to Use Proxies to Rotate IP Addresses In Python

You can scrape a list of free proxies and save them in a text file named list_proxy.txt. You can then follow the steps below to rotate IP addresses in Python.

Import Libraries

You must import the Python’s requests module using the command below.

import requests

If the module is not installed in your environment, you can install it using the below command.

!pip install requests

Create A Function

You must create a send_request function that will take two parameters, i-e, session and proxy. It would be better to use an HTTP proxy as most free proxies don’t use the HTTPS protocol. We will use try-except blocks because most free proxies don’t work as they have been blocklisted.

def send_request(session, proxy):
   try:
       response = session.get('http://httpbin.org/ip', proxies={'http': f"http://{proxy}"})
       print(response.json())
   except:
       pass

Read The .txt File

Here comes the main section of the code. You have to read your list_proxy.txt file and save it to the variable named proxies as shown below.

if __name__ == "__main__":
   with open('list_proxy.txt', 'r') as file:
       proxies = file.readlines()

Use a For Loop

You have to use the requests module to create a session in Python. We used a for loop to loop through the proxy in the code below. We then passed the session and the proxy to the send_request function. 

with requests.Session() as session:
   for proxy in proxies:
       send_request(session, proxy)

Run the Script

You can run the script and see the output as shown below. You will get a long list of proxies but remember, they won’t work. 

python rotateproxy.py

Your whole code will be as:

import requests


def send_request(session, proxy):
   try:
       response = session.get('http://httpbin.org/ip', proxies={'http': f"http://{proxy}"})
       print(response.json())
   except:
       pass


if __name__ == "__main__":
   with open('list_proxy.txt', 'r') as file:
       proxies = file.readlines()

   with requests.Session() as session:
       for proxy in proxies:
           send_request(session, proxy)

Tips To Build A Proxy Rotator

Given below are some tips to rotate proxies in Python when using a rotating proxy.

Use Reliable, Free Proxy Services

Most of the available free proxy services are congested, which leads to unexpected crashes and frustrating delays. Therefore, when you plan to go for free proxies, you have to check if they match your scraping needs without causing any significant drawdowns.

Most free proxies are valid for a limited time, so you can build your own logic in Python that can rotate the free proxy list rotator with working IP addresses. This way, you won’t get any disruptions during web scraping. 

Refrain From Using Predictable Proxy IP Addresses

You have to avoid using IP addresses that follow a particular sequence or a predictable format like belonging to the same group. It is because most anti-scraping tools can easily detect the requests sent from such IP addresses. For instance, you should abstain from using the following sequence of rotating IP addresses as it will quickly raise the red flag.

103.243.132.11

103.243.132.12

103.243.132.13

103.243.132.14

Consider A Premium Proxy Service

The free proxy IP rotators are insecure and slow. So, using them will not be suitable if you perform advanced and large-scale web scraping projects. Using good premium proxies, you can get a high-quality service and scrape thousands of web pages without experiencing blockades or disruptions. 

To use a premium proxy service, you will have to pay a few dollars and get a provider that will meet your data extraction needs and guarantee your privacy.

Go For Elite Proxies

Given below are three main categories of proxies on the Internet.

• Elite Proxies
• Anonymous Proxies
• Transparent Proxies

Elite proxies are the best option to consider among these proxies as they can prevent detections and bypass restrictions. You can only send the REMOTE_ADDR header using an elite proxy while keeping the other headers empty. This way, you can achieve optimal privacy with these proxies.

On the other hand, a transparent proxy reveals your IP address details and sends your real IP via the HTTP_VIA header and HTTP_X_FORWARDED_FOR header. The anonymous proxy does not disclose your actual IP address. It either dispatches the proxy’s IP or leaves it empty.

Combine IP Rotation With User Agent Rotation

You can use IP rotation for rotating multiple IP addresses and avoiding detection. However, some anti-scraping measures can identify such activities and block you from harvesting data. Therefore, you also need to rotate user agents apart from rotating IP addresses to increase your chances of success. 

FAQs:

Conclusion

It is simple to create a proxy rotator in Python. You can generate random proxies and then build logic to scrape data from websites. Apart from datacenter and residential proxies, the other categories are elite, anonymous, and transparent proxies. Elite proxies are the best option because they are hard to be detected by the target website. You can use anonymous proxies if you just want to keep your privacy on the Internet. Lastly, you can use transparent proxies, but they offer the lowest chance of success. This article hopes to give in-depth information and give examples of how to use proxies to rotate IP addresses in python with simple coding steps