Virtualenv is a tool to create isolated Python environments.It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments. Beautiful Soup is a Python library for pulling data out of HTML and XML files. The followings steps are to be followed for creating virtualenv and installing BeautifulSoup4 module in it.
~$ mkdir virtual ~$ cd virtual ~/virtual$ pip install virtualenv Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/lib/python2.7/dist-packages Cleaning up... ~/virtual$ virtualenv vir1 New python executable in vir1/bin/python Installing distribute..................................................................................................................... . .......................................................................done. Installing pip...............done. ~/virtual$ source vir1/bin/activate (vir1)user@ubuntu:~/virtual$ pip install BeautifulSoup4 Downloading/unpacking BeautifulSoup4 Downloading beautifulsoup4-4.2.1.tar.gz (139Kb): 139Kb downloaded Running setup.py egg_info for package BeautifulSoup4 Installing collected packages: BeautifulSoup4 Running setup.py install for BeautifulSoup4 Successfully installed BeautifulSoup4 Cleaning up...
task
The task was to create a virtualenv and write a file planetparser.py which when executed as ./planetparser.py would print the blog title and author names from http://planet.fedoraproject.org/. The required modules are to be installed in the virtualenv.
We use urllib2 module for opening the url and Beautifulsoup module for parsing through the webpage and retrieving required contents.
link to the code link
CODE
#!/usr/bin/env python
from bs4 import BeautifulSoup #imports BeautifulSoup package used for for parsing through the webpage and retrieving required contents.
import urllib2 # imports urllib2 required to open the webpage corresponding through the url passed to it.
import sys
def Planet_parser():
url="http://planet.fedoraproject.org/" #url of the webpage to be scrapped.
contents=urllib2.urlopen(url).read() #opens the url and reads its contents
soup=BeautifulSoup(contents) #passes the contents to BeautifulSoup which returnsSoup object.
soup.prettify() # represents the source page document as a nested data structure
Author=soup.find_all('div', {'class':'blog-entry-author'}) # finds div tags belonging to class=blog-entry-author and stores it in Author as a navigable strings.
Title=soup.find_all('div',{'class':'blog-entry-title'}) #finds div tags belonging to class=blog-entry-title and stores itr in Title as navigable strings.
count1= len(Author) # counts the no of div tags in Author
count2= len(Title) #counts the no of div tags in Title
i=0 #initialisation
while i<count1:
title1=BeautifulSoup(str(Title[i])) #passes the ith navigable string of Title to BeautifulSoup for parsing.
author1= BeautifulSoup(str(Author[i])) #passes the ith navigable string of Author to BeautifulSoup for parsing.
print "title of the blog :",title1.get_text() # prints title of the blog
print "author of the blog:",author1.a.get_text() #prints author of the blog
i=i+1 #incrementation
if __name__ == '__main__':
Planet_parser() #calls planet_parser function
sys.exit(1) #exits
To create a greet command which will print 'Hi' and a stock command which return the current share value.
The code can be run by :-
$ python psh.py
or, if made executable
$ ./psh.py
from cmd2 import Cmd #cmd2 module is imported for making command
import requests #requests module is imported for http requests
import getpass
__version__ = '0.1'
class Application(Cmd):
def __init__(self):
Cmd.__init__(self)
def do_hello(self, line): # hello command is made which will print "Hello: line"
print "Hello:", line
def do_sayit(self, line): # sayit command is made which will print "python rocks"
print "Python rocks!"
def do_greet(self, line): # greet command is made which will print hi
x = getpass.getuser()
print "hi %s" % x
def do_stock(self, line): # stock command is made which will print sharevalue
try:
r = requests.get('http://download.finance.yahoo.com/d/quotes.csv?s='+line+'&f=l1')
b = float(r.text)
if b == 0.00:
print "Invalid Nasdaq Code"
else:
print "%f" % (b)
except:
print "Connection Problem"
if __name__ == '__main__':
app = Application()
app.cmdloop()
click here for the link of actual code.
Write a file sharevalue.py such that when we run the command ./sharevalue.py <NASDAQ_symbol_of_company> will give the share value of the company.
In this code, we import urllib2 package.The urllib2 module defines functions and classes which help in opening URLs (mostly HTTP) in a complex world — basic and digest authentication, redirections, cookies and more.
share_value Function
Nasdaq symbol of the company is passed to this function which inserts it in the url of yahoo sharevalue finder. urllib2.open(url) function is used to open the url for finding the sharevalue. print_sharevalue() function is then called from this function.
print_sharevalue Function This function as the name suggests reads the sharevalue file and prints the sharevalue after checking if the NASDAQ symbol is valid. The is done by checking if the the sharevalue read is equal to 0.0 . If sharevalue is zero , it indicates that the NASDAQ symbol is invalid, hence prints "invalid nasdaq symbol" . If nasdaq value is valid it prints the corresponding sharevalue.
link to the code link
To create a greet command which will print 'Hi' and a stock command which return the current share value.
The code can be run by :-
$ python psh.py
or, if made executable
$ ./psh.py
from cmd2 import Cmd #cmd2 module is imported for making command
import requests #requests module is imported for http requests
__version__ = '0.1'
class Application(Cmd):
def __init__(self):
Cmd.__init__(self)
def do_hello(self, line): # hello command is made which will print "Hello: line"
print "Hello:", line
def do_sayit(self, line): # sayit command is made which will print "python rocks"
print "Python rocks!"
def do_greet(self, line): # greet command is made which will print hi
print "hi"
def do_stock(self, line): # stock command is made which will print sharevalue
try:
r = requests.get('http://download.finance.yahoo.com/d/quotes.csv?s='+line+'&f=l1')
b = float(r.text)
if b == 0.00:
print "Invalid Nasdaq Code"
else:
print "%f" % (b)
except:
print "Connection Problem"
if __name__ == '__main__':
app = Application()
app.cmdloop()
click here for the link of actual code.
In this assignment. we will create a virtualenv and run this script which will create a small shell like environment with 'Cmd' as prompt, and where we can give some commands to it.
The code for the problem can be found here
The solution to the above problem is given below.
from cmd2 import Cmd import requests import pwd __version__ = '0.1' class Application(Cmd): """ The main Application class """ def __init__(self): Cmd.__init__(self) def do_hello(self, line): # defining the 'hello' command print "Hello:", line # executing the instructions for the command def do_sayit(self, line): # defining the 'sayit' command print "Python rocks!" # executing the instructions for the command def do_stock(self, line): # defining the 'stock' command nasdaq = raw_input("Enter NASDAQ code: ") # taking the NASDAQ code as input from the user url = "http://download.finance.yahoo.com/d/quotes.csv?s=%s&f=l1" \ % (nasdaq) # the recquired URL from which the share price shall be obtained value = requests.get(url).text # storing the share price from the URL in 'value' if value == 0: # incorrect NASDAQ code provided print "Incorrect NASDAQ symbol." # generate error message else: # share price has been obtained print "The current share value is ", value # display the value def do_greet(self, line): # defining the 'greet' command total_list = pwd.getpwall() # a list of all password database entries for x in total_list: # iterating through the list if x[5].find('home') == 1: # the entry for the user has been found print "Hello %s" % x[0] # print the user name break # exit from the loop if __name__ == '__main__': app = Application() app.cmdloop()
To run the program, save it as psh.py, and:
$ python psh.py
To write a program that takes NASDAQ index value from a web page and prints the value of the index to the screen.
$python sharevalue.py (index) f.e $python sharevalue.py GOOG
To create a python shell which will greet and give the current stock value.
#!/usr/bin/env python
from cmd2 import Cmd
from getpass import getuser
import sys
import requests
__version__ = '0.1'
class Application(Cmd):
def __init__(self):
Cmd.__init__(self)
def do_greet(self,line):
print "Hi! %s " %(getuser()) # getuser()will give the username of the system as output.
def do_stock(self,line):
link=requests.get('http://download.finance.yahoo.com/d/quote.csv?s='+line+'&f=l1') # opens the link with <line> which should be a valid NASDAQ symbol.
print link.text # prints the output of the link.
if __name__ == '__main__':
app = Application()
app.cmdloop()
To go to program click here.
To run this requests and cmd2 modules should be installed in the virtual env.
[Shantanu@dhcppc0] myshell $ (master) source ~/virt1/bin/activate (virt1)[Shantanu@dhcppc0] myshell $ (master) python myshell.py
(Cmd) greet Hi! Shantanu (Cmd) stock GOOG 910.68
Assignment is to write a program that constitutes the following two commands:
1)greet : which will say "Hi" to the user. 2)stock : which will print the cureent stock value.
//These comands are to be executed using the requests library.
#!/usr/bin/env python
from cmd2 import Cmd
from getpass import getuser
import sys
import requests
__version__ = '0.1'
class Application(Cmd):
def __init__(self):
Cmd.__init__(self)
def do_greet(self,line):
print "Hi! %s " %(getuser())
def do_stock(self,line):
link=requests.get('http://download.finance.yahoo.com/d/quote.csv?s='+line+'&f=l1')
result=link.text
print result
if __name__ == '__main__':
app = Application()
app.cmdloop()
code is here
$python shell.py
(virt1)[root@Avinash avinash]# python shell.py (Cmd) greet Hi! avinash (Cmd) stock GOOG 910.68
I installed requests and cmd2 modules for this assignment in my 'virt1' environment.
(virt1) $ yolk -l Python - 2.7.5 - active development (/usr/lib/python2.7/lib-dynload) beautifulsoup4 - 4.2.1 - active cmd2 - 0.6.5.1 - active lxml - 3.2.1 - active pip - 1.3.1 - active requests - 1.2.3 - active setuptools - 0.6c11 - active wsgiref - 0.1.2 - active development (/usr/lib/python2.7) yolk - 0.4.3 - active $ python myshellv1.py
A link to the source code.
(Cmd) greet Hi, m0rin09ma3 (Cmd) stock GOOG 910.68
In the Application class, there are 2 methods (do_greet() for greeting, do_stock() for getting quotes from specified url)
def do_greet(self, line): print "Hi, %s" % os.getlogin() def do_stock(self, line): quote = {'s': line, 'f': 'l1'} target_url = 'http://download.finance.yahoo.com/d/quotes.csv' r = requests.get(target_url, params=quote) #print r.url print r.text
To create a code to print "Hi, <username>" when greet command is given and to print the current stock value when stock command is given.
from cmd2 import Cmd
from getpass import getuser
import requests
import sys
__version__ = '0.1'
class Application(Cmd):
""" The main Application class """
def __init__(self):
Cmd.__init__(self)
def do_hello(self, line):
print "Hello:", line
def do_sayit(self,line):
print "Python Rocks!"
def do_greet(self,line):
print "hi,%s" % (getuser())
def do_stockgoog(self,line):
a = requests.get("http://download.finance.yahoo.com/d/quotes.csv?s=GOOG&f=l1")
value=a.text
print value
if __name__ == '__main__':
app = Application()
app.cmdloop()