{"id":863,"date":"2021-01-28T00:16:34","date_gmt":"2021-01-27T16:16:34","guid":{"rendered":"https:\/\/byy3.com\/?p=863"},"modified":"2021-01-28T02:34:20","modified_gmt":"2021-01-27T18:34:20","slug":"product-analysis-using-web-scraping-technique-in-python-open-code","status":"publish","type":"post","link":"https:\/\/byy3.com\/?p=863","title":{"rendered":"Product Analysis using Web Scraping Technique in Python open code"},"content":{"rendered":"
\n
\n
\n
\n
\n

Web Scraping is one of the Data Scraping technique in which data is extracted from the websites for analysis.In this project we will learn the how to analyze the product in an online shop like flipkart, for example we will analyze various brands of Mobile Tablets sold in the flipkart web site and suggest the medium range product in price range.<\/p>\n

Using the web scraping techniques we will be able to get the details prices,specifications, reviews,highlights and ratings for any product in the website.<\/p>\n

We will be using the python modules urlopen(from urllib library) and BeautifulSoup(from bs4 library).<\/p>\n

\n

import bs4
\nimport pandas as pd
\nimport numpy as np
\nimport seaborn as sns
\nimport matplotlib.pyplot as plt
\nfrom urllib.request import urlopen as uReq
\nfrom bs4 import BeautifulSoup as soup<\/p>\n<\/blockquote>\n

Fir<\/span>st we need to read the contents of the web page which displays the search results of the product we are going to analyse ,the tablet and parse the html content using the BeautifulSoup module.<\/p>\n

\n

myurl = \u201chttps:\/\/www.flipkart.com\/tablets\/pr?sid=tyy,hry&marketplace=FLIPKART<\/a>\"
\nuclient = uReq(myurl)
\npage_html = uclient.read()
\nuclient.close()<\/p>\n

psoup = soup(page_html, \u201chtml.parser\u201d)<\/p>\n<\/blockquote>\n

Below is the screen short of the web page we are going to get the data for analysis,right click on the page and do inspect, it will show the HTML document tags for each element in the page.<\/p>\n

\n
\n
\n
\n
\n
\"Product<\/div>\n

\"Product<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/figure>\n

Right click on the Page index tab in the page select the inspect option and search for the elements listing and find the class name for that element and use it for getting the href of the link to the search result pages for all the product listed,as show in the picture below<\/p>\n

\n
\n
\n
\n
\n
\"Product<\/div>\n

\"Product<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/figure>\n

Now find all the page references and store it in a list,as detailed in the code below<\/p>\n

\n

page_urls = list()
\nfor containers in psoup.findAll(\u2018div\u2019,{\u2018class\u2019:\u2019_2MImiq\u2019}):
\na_list = containers.findAll(\u2018a\u2019,{\u2018class\u2019:\u2019ge-49M\u2019})<\/p>\n

#uncomment and use the below 2 lines to get all the search 10 pages
\n#for a in a_list:
\n# page_urls.append(\u2018https:\/\/www.flipkart.com\u2019+<\/a>\u00a0a[\u2018href\u2019])<\/p>\n

#use the below 2 lines to get single search page
\na= a_list[0]
\npage_urls.append(\u2018https:\/\/www.flipkart.com'+<\/a>\u00a0a[\u2018href\u2019])<\/p><\/blockquote>\n

Once we get the page url link, we need to inspect each product listed in the page as below<\/p>\n

\n

for url in page_urls:
\nprint(url)
\nuclient = uReq(url)
\npage_html = uclient.read()
\nuclient.close()
\npsoup = soup(page_html, \u201chtml.parser\u201d)<\/p>\n

prod_urls = list()
\nfor containers in psoup.findAll(\u2018div\u2019,{\u2018class\u2019:\u2019_13oc-S\u2019}):
\nfor a in containers:
\na_list = a.findAll(\u2018a\u2019,{\u2018class\u2019:\u2019_1fQZEK\u2019})
\n#print(a_list[\u2018href\u2019])
\n#prod_urls.append(\u2018https:\/\/www.flipkart.com\u2019+<\/a>\u00a0a_list[\u2018href\u2019])<\/p>\n

for a in a_list:
\nprod_urls.append(\u2018https:\/\/www.flipkart.com'+<\/a>\u00a0a[\u2018href\u2019])<\/p>\n

for p_url in prod_urls:
\nuclient = uReq(p_url)
\npage_html = uclient.read()
\nuclient.close()
\npsoup = soup(page_html, \u201chtml.parser\u201d)<\/p>\n

all_procuct_items = psoup.find(\u2018div\u2019, attrs={\u2018class\u2019 : \u2018_3k-BhJ\u2019})
\nall_procuct_items = all_procuct_items.findAll(\u2018tr\u2019, attrs={\u2018class\u2019 : \u2018_1s_Smc row\u2019})<\/p>\n

#container variable contains the html of product title which is stored in div tag and class is \u201c_1AtVbE col-12\u201312\u201d
\ncontainer= psoup.findAll(\u201cdiv\u201d,{\u201cclass\u201d:\u201d_1AtVbE col-12\u201312\"})
\nfor product_item in container:
\nproduct_dict = dict()
\nrating_dict= dict()
\nprice_dict = dict()
\nbrandname_dict = dict()
\nn = product_item.findAll(\u201cspan\u201d,{\u201cclass\u201d:\u201dB_NuCI\u201d})
\np = product_item.findAll(\u201cdiv\u201d,{\u201cclass\u201d:\u201d_30jeq3 _16Jk6d\u201d})<\/p>\n

r = product_item.findAll(\u201cdiv\u201d,{\u201cclass\u201d:\u201d_2d4LTz\u201d})
\nfor i in n:
\nproduct_dict[\u2018name\u2019] = i.text
\nstrtmp = i.text.split(\u201c \u201c)
\nbrandname_dict[\u2018brandname\u2019] = strtmp[0]
\nbrandnames_list.append(strtmp[0])
\nproducts_list.append(i.text)
\nfor j in p:
\njStr = j.text
\njStr = jStr.replace(\u201c\u20b9\u201d, \u201c\u201d)
\njStr = jStr.replace(\u201c,\u201d, \u201c\u201d)
\nprice_dict[\u2018price\u2019] = int( jStr)
\nprices_list.append(int( jStr))
\nfor k in r:
\n#print (i.text)
\nrating_dict[\u2018rating\u2019] = k.text
\nratings_list.append(k.text)<\/p><\/blockquote>\n

We can now export data extracted for the details of the products to the csv file using the code below<\/p>\n

\n

df = pd.DataFrame({\u2018Brand\u2019:brandnames_list,\u2019Price\u2019:prices_list,\u2019Ratings\u2019:ratings_list,\u2019ProductName\u2019:products_list})
\ndf.to_csv (\u2018export_products_dataframe.csv\u2019)<\/p>\n<\/blockquote>\n

We can use the box plot ,cat plot and bar plot available in the seaborn module for visualizing the price range of the product as shown in the pictures below,using the code below<\/p>\n

\n

df[\u2018Price\u2019] = df[\u2018Price\u2019].astype(np.float)
\nsns.boxplot(x=df[\u2018Price\u2019])<\/p>\n

sns.catplot(x = \u201cPrice\u201d, # x variable name
\ny = \u201cBrand\u201d, # y variable name
\nhue = \u201cRatings\u201d, # group variable name
\ndata = df, # dataframe to plot
\nkind = \u201cbar\u201d)<\/p>\n

df.groupby(\u2018Brand\u2019).plot(x=\u2019Brand\u2019, y=\u2019Price\u2019)<\/p>\n

sns.barplot(x = \u2018Brand\u2019,
\ny = \u2018Price\u2019,
\ndata = df)<\/p>\n<\/blockquote>\n

\n
\n
\n
\n
\n
\"Product<\/div>\n

\"Product<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>

Product listing based on ratings and prices<\/figcaption><\/figure>\n
\n
\n
\n
\n
\n
\"Product<\/div>\n

\"Product<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>

Price Range chart<\/figcaption><\/figure>\n
\n
\n
\n
\n
\n
\"Product<\/div>\n

\"Product<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>

Price range chart for the Brand Samsung<\/figcaption><\/figure>\n
\n
\n
\n
\n
\n
\"Product<\/div>\n

\"Product<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>

Price range chart for the Brand Apple<\/figcaption><\/figure>\n<\/div>\n<\/div>\n<\/section>\n<\/div>\n<\/article>\n
\n
\n
\n
\n
\n
\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n
\n
\n
\n
\n

AI Enthusiast<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\n
\n
\n
\n
\n
Sep 1, 2020<\/div>\n<\/div>\n
Overloading in Python<\/a><\/div>\n
<\/div>\n
\n
\n

In Object Oriented programming approach the concept of polymorphism is about having a class objects in many forms. Polymorphism is of two types one is static polymorphism and the other is dynamic polymorphism.<\/p>\n

\n
\n
\n
\n
\"Product<\/div>\n

\"Product<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/figure>\n

Python do not have compile time polymorphism directly, that is because python programs do not use compilation specific to the machine in which they run ,instead the python programs are compiled to be run on a python virtual machine, and hence compilation is not specific to the machine, which makes python portable across platforms,more easier to learn and machine independent ,the picture below gives an overview of the components used to convert a python language program to a running code. \u2026<\/p>\n<\/section>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
\r\nimport bs4<\/span>\r\nimport pandas as pd<\/span>\r\nimport numpy as np<\/span>\r\nimport seaborn as sns<\/span>\r\nimport matplotlib.pyplot as plt<\/span>\r\nfrom scipy import stats<\/span>\r\nimport plotly_express as px<\/span>\r\nfrom urllib.request import urlopen as uReq<\/span>\r\nfrom bs4 import BeautifulSoup as soup\r\n<\/span>\r\nmyurl = \"https:\/\/www.flipkart.com\/tablets\/pr?sid=tyy,hry&marketplace=FLIPKART\"<\/span>\r\nuclient = uReq(myurl)<\/span>\r\npage_html = uclient.read()<\/span>\r\nuclient.close()<\/span>\r\n\r\npsoup = soup(page_html, \"html.parser\")<\/span>\r\n\r\nproducts_list = list()<\/span>\r\nprices_list = list()<\/span>\r\nratings_list= list()<\/span>\r\nmodelnames_list= list()<\/span>\r\nbrandnames_list = list()<\/span>\r\n\r\nrams_list= list()<\/span>\r\ncolors_list= list()<\/span>\r\ndisplays_list= list()<\/span>\r\n\r\nconnectivities_list= list()<\/span>\r\n\r\npage_urls = list()<\/span>\r\nfor containers in psoup.findAll('div',{'class':'_2MImiq'}):<\/span>\r\n page_href_list = containers.findAll('a',{'class':'ge-49M'})<\/span>\r\n #uncomment the below 2 lines to get all the search 10 pages<\/span>\r\n #for a in a_list:<\/span>\r\n # page_urls.append('https:\/\/www.flipkart.com'+ a['href'])<\/span>\r\n #uncomment the below 2 lines to get single search page<\/span>\r\n page_href= page_href_list[0]<\/span>\r\n page_urls.append('https:\/\/www.flipkart.com'+ page_href['href'])<\/span>\r\n\r\nfor url in page_urls:<\/span>\r\n print(url)<\/span>\r\n uclient = uReq(url)<\/span>\r\n page_html = uclient.read()<\/span>\r\n uclient.close()<\/span>\r\n psoup = soup(page_html, \"html.parser\")<\/span>\r\n prod_urls = list()<\/span>\r\nfor containers in psoup.findAll('div',{'class':'_13oc-S'}):<\/span>\r\n for a in containers:<\/span>\r\n a_list = a.findAll('a',{'class':'_1fQZEK'})<\/span>\r\n #print(a_list['href'])<\/span>\r\n #prod_urls.append('https:\/\/www.flipkart.com'+ a_list['href'])<\/span>\r\n  for a in a_list:<\/span>\r\n prod_urls.append('https:\/\/www.flipkart.com'+ a['href'])<\/span>\r\n\r\n for p_url in prod_urls:<\/span>\r\n uclient = uReq(p_url)<\/span>\r\n page_html = uclient.read()<\/span>\r\n uclient.close()<\/span>\r\n psoup = soup(page_html, \"html.parser\")<\/span>\r\n all_procuct_items = psoup.find('div', attrs={'class' : '_3k-BhJ'})<\/span>\r\n all_procuct_items = all_procuct_items.findAll('tr', attrs={'class' : '_1s_Smc row'})<\/span>\r\n for procuct_item in all_procuct_items:<\/span>\r\n dataname = procuct_item.findAll(\"td\", {\"class\": \"_1hKmbr col col-3-12\"})<\/span>\r\n datadetails = procuct_item.findAll(\"td\", {\"class\": \"URwL2w col col-9-12\"})<\/span>\r\n \"\"\"\r\n color = dict()<\/span>\r\n display = dict()<\/span>\r\n modelname = dict()<\/span>\r\n connectivity = dict()<\/span>\r\n\r\n for names in dataname:<\/span>\r\n if( 'Name' in names.text):<\/span>\r\n for details in datadetails:<\/span>\r\n modelname['modelname'] = details.text<\/span>\r\n modelnames.append(details.text)<\/span>\r\n elif( 'Color' in names.text):<\/span>\r\n for details in datadetails:<\/span>\r\n color['color'] = details.text<\/span>\r\n colors.append(details.text)<\/span>\r\n elif(\"Display\" in names.text):<\/span>\r\n for details in datadetails:<\/span>\r\n display['display'] = details.text<\/span>\r\n displays.append(details.text)<\/span>\r\n elif(\"Connectivity\" in names.text):<\/span>\r\n for details in datadetails:<\/span>\r\n connectivity['connectivity'] = details.text<\/span>\r\n connectivities.append(details.text)<\/span>\r\n \"\"\"<\/span>\r\n container= psoup.findAll(\"div\",{\"class\":\"_1AtVbE col-12-12\"})<\/span>\r\n #container variable contains the html of product title which is stored in div tag and class is \"_1AtVbE col-12-12\"<\/span>\r\n\r\n for product_item in container:<\/span>\r\n product_dict = dict()<\/span>\r\n rating_dict= dict()<\/span>\r\n price_dict = dict()<\/span>\r\n brandname_dict = dict()<\/span>\r\n n = product_item.findAll(\"span\",{\"class\":\"B_NuCI\"})<\/span>\r\n p = product_item.findAll(\"div\",{\"class\":\"_30jeq3 _16Jk6d\"})<\/span>\r\n  r = product_item.findAll(\"div\",{\"class\":\"_2d4LTz\"})<\/span>\r\n for i in n:<\/span>\r\n product_dict['name'] = i.text<\/span>\r\n strtmp = i.text.split(\" \")<\/span>\r\n brandname_dict['brandname'] = strtmp[0]<\/span>\r\n brandnames_list.append(strtmp[0])<\/span>\r\n products_list.append(i.text)<\/span>\r\n for j in p:<\/span>\r\n jStr = j.text<\/span>\r\n jStr = jStr.replace(\"\u20b9\", \"\")<\/span>\r\n jStr = jStr.replace(\",\", \"\")<\/span>\r\n price_dict['price'] = int( jStr)<\/span>\r\n prices_list.append(int( jStr))<\/span>\r\n for k in r:<\/span>\r\n #print (i.text)<\/span>\r\n rating_dict['rating'] = k.text<\/span>\r\n ratings_list.append(k.text)<\/span>\r\n\r\nprint (\"Number of items in the products list = \", len(products_list))<\/span>\r\nprint (\"Number of items in the prices list = \", len(prices_list))<\/span>\r\nprint (\"Number of items in the ratings list = \", len(ratings_list))<\/span>\r\n\r\ndf = pd.DataFrame({'Brand':brandnames_list,'Price':prices_list,'Ratings':ratings_list,'ProductName':products_list})<\/span>\r\ndf.to_csv ('export_products_dataframe.csv')<\/span>\r\n\r\ndf['Price'] = df['Price'].astype(np.float)<\/span>\r\nsns.boxplot(x=df['Price'])<\/span>\r\n\r\nsns.catplot(x = \"Price\", # x variable name<\/span> \r\n y = \"Brand\", # y variable name<\/span>\r\n hue = \"Ratings\", # group variable name<\/span>\r\n data = df, # dataframe to plot\r\n<\/span> kind = \"bar\")<\/span>\r\ndf.groupby('Brand').plot(x='Brand', y='Price')<\/span>\r\nsns.barplot(x = 'Brand', y = 'Price', data = df)<\/span><\/pre>\n","protected":false},"excerpt":{"rendered":"
Web Scraping is one of the Data Scraping technique in w […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[402,401],"class_list":["post-863","post","type-post","status-publish","format-standard","hentry","category-python","tag-python-scrapy","tag-scrapy"],"_links":{"self":[{"href":"https:\/\/byy3.com\/index.php?rest_route=\/wp\/v2\/posts\/863","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/byy3.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/byy3.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/byy3.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/byy3.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=863"}],"version-history":[{"count":0,"href":"https:\/\/byy3.com\/index.php?rest_route=\/wp\/v2\/posts\/863\/revisions"}],"wp:attachment":[{"href":"https:\/\/byy3.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=863"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/byy3.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=863"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/byy3.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}