Fix Amazon Scraper 503 – Dr. Trippers Stuff

I am getting a 503 error on this: import time
import random
import requests
from bs4 import BeautifulSoup

# Define the search query and headers for the HTTP request
query = “Nike Men’s Hoodie size M”
headers = {“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36″}

# Send the HTTP request and parse the HTML response using BeautifulSoup
url = f”https://www.amazon.com/s?k={query}”
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, “html.parser”)

# Find all product listings on the page and extract their prices
prices = []
for item in soup.find_all(“span”, class_=”a-price”):
price = float(item.find(“span”, class_=”a-offscreen”).text[1:])
prices.append(price)

# Print the high and low selling price for the product
if len(prices) > 0:
print(f”High Selling Price: ${max(prices):.2f}”)
print(f”Low Selling Price: ${min(prices):.2f}”)
else:
print(“No prices found for this product.”)
print(response)
# Add a random delay between 1 and 5 seconds to avoid triggering rate limits or blocking by the server
time.sleep(random.uniform(1, 5))

A 503 error indicates that the server is temporarily unavailable, which could be due to server overload, maintenance, or blocking your request. To address this issue, you can try the following steps:

1. **Add Retry Logic**: Implement retry logic to handle transient errors.
2. **Check User-Agent**: Make sure the User-Agent string is not blocked.
3. **Respect Rate Limits**: Ensure delays between requests to avoid being blocked.

Here is an improved version of your script with retry logic and better handling of HTTP errors:

“`python
import time
import random
import requests
from bs4 import BeautifulSoup

def fetch_url(url, headers, retries=3):
for i in range(retries):
try:
response = requests.get(url, headers=headers)
if response.status_code == 200:
return response
elif response.status_code == 503:
print(“Server is unavailable. Retrying…”)
time.sleep(random.uniform(1, 5))
except requests.RequestException as e:
print(f”Request failed: {e}”)
time.sleep(random.uniform(1, 5))
return None

# Define the search query and headers for the HTTP request
query = “Nike Men’s Hoodie size M”
headers = {
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36″
}

# Send the HTTP request and parse the HTML response using BeautifulSoup
url = f”https://www.amazon.com/s?k={query.replace(‘ ‘, ‘+’)}”
response = fetch_url(url, headers)

if response:
soup = BeautifulSoup(response.content, “html.parser”)

# Find all product listings on the page and extract their prices
prices = []
for item in soup.find_all(“span”, class_=”a-price”):
try:
price = float(item.find(“span”, class_=”a-offscreen”).text[1:])
prices.append(price)
except (AttributeError, ValueError) as e:
print(f”Error parsing price: {e}”)

# Print the high and low selling price for the product
if len(prices) > 0:
print(f”High Selling Price: ${max(prices):.2f}”)
print(f”Low Selling Price: ${min(prices):.2f}”)
else:
print(“No prices found for this product.”)
else:
print(“Failed to fetch the URL after multiple retries.”)
“`

This script:
1. Retries the request up to three times if a 503 error is encountered.
2. Adds error handling for requests and price parsing.
3. Sleeps for a random interval between 1 and 5 seconds to avoid rate limiting.
Failed. DOes amazon have an API?
Yes, Amazon has an official API called the **Amazon Product Advertising API**. This API allows you to programmatically retrieve product details, including pricing, images, descriptions, and more. Here’s a brief overview of how to use it:

### Steps to Use Amazon Product Advertising API

1. **Sign Up for an AWS Account**:
– Create an AWS account if you don’t have one.
– Sign in to the [AWS Management Console](https://aws.amazon.com/).

2. **Sign Up for the Amazon Product Advertising API**:
– Go to the [Amazon Associates Program](https://affiliate-program.amazon.com/) and sign up.
– Apply for the Product Advertising API.

3. **Get Your Access Keys**:
– After approval, go to the [AWS IAM Console](https://console.aws.amazon.com/iam/).
– Create a new IAM user with programmatic access.
– Note down the Access Key ID and Secret Access Key.

4. **Install Boto3 and Setup Credentials**:
– Install the Boto3 library for accessing AWS services.
– Configure your AWS credentials.

“`sh
pip install boto3
“`

5. **Make API Requests**:
– Use the Boto3 library to make requests to the Amazon Product Advertising API.

Here is an example script to get product details using Boto3:

“`python
import boto3
from botocore.exceptions import NoCredentialsError, PartialCredentialsError

# Configure AWS credentials (ensure your credentials are set up in ~/.aws/credentials)
# [default]
# aws_access_key_id = YOUR_ACCESS_KEY
# aws_secret_access_key = YOUR_SECRET_KEY

# Create a client for the Product Advertising API
client = boto3.client(
“advertising”,
region_name=”us-east-1″, # specify your region
aws_access_key_id=”YOUR_ACCESS_KEY”,
aws_secret_access_key=”YOUR_SECRET_KEY”,
api_version=’v1′
)

# Define the search parameters
params = {
“Keywords”: “Nike Men’s Hoodie size M”,
“SearchIndex”: “Fashion”,
“Resources”: [
“ItemInfo.Title”,
“Offers.Listings.Price”
]
}

try:
# Make the API request
response = client.search_items(**params)
items = response.get(“SearchResult”).get(“Items”)

# Extract prices
prices = []
for item in items:
title = item.get(“ItemInfo”).get(“Title”).get(“DisplayValue”)
price = item.get(“Offers”).get(“Listings”)[0].get(“Price”).get(“Amount”)
prices.append(price)
print(f”Title: {title}, Price: ${price:.2f}”)

# Print the high and low selling price for the product
if len(prices) > 0:
print(f”High Selling Price: ${max(prices):.2f}”)
print(f”Low Selling Price: ${min(prices):.2f}”)
else:
print(“No prices found for this product.”)
except (NoCredentialsError, PartialCredentialsError):
print(“Credentials not available or partially configured.”)
except Exception as e:
print(f”Error: {e}”)
“`

This example demonstrates how to configure and use the Amazon Product Advertising API to search for products and retrieve their prices. Be sure to replace `”YOUR_ACCESS_KEY”` and `”YOUR_SECRET_KEY”` with your actual AWS credentials.

### Note
– The Product Advertising API has usage limits, and you need to follow Amazon’s policies to avoid being blocked.
– You must be an Amazon Associate to use the Product Advertising API.

For more detailed information, you can refer to the [Amazon Product Advertising API Documentation](https://webservices.amazon.com/paapi5/documentation/).