Thank you for visiting!
My little window on internet allowing me to share several of my passions
Categories:
- OpenBSD
- FreeBSD
- Nas
- DragonflyBSD
- fapws
- Alpine Linux
- OpenBSD
- Openbox
- Desktop
- Security
- nvim
- yabitrot
- nmctl
- Tint2
- Firewall
- vdcron
- VPN
- Project Management
- Hifi
- Alarm
Most Popular Articles:
Last Articles:
Light webserver
Posted on 2017-05-01 14:49:00 from Vincent in OpenBSD fapws
In this post I would like to share some feedback concerning the Fast Asynchronous Python Web Server I'm using for this web site.
Fapws is no more updated since 2012 !!! Should I migrate to something better: httpd ?
Fapws is a very simple and light webserver following the WSGI rules.
This web server is existing since long time, but I'm not sure if it's often used. In my cases, it does the job and it runs on some of my OpenBSD machines since few years now. Mainly the one managing my home alarm system. At that time, I've selected it because it's very light and single thread. It was important for me that the webserver does not reserve too much resources and, in the worst case, block the other processes running on this small machine.
All in all, Fapws does the job, but I've observed few crashes.
By looking a the github history, Fapws is quite old and not really maintained by the author.
After recent contacts with William, the initial author, I've decided to look myself why I have those crashes.
It was a nice investigation for me because I'm not a C programmer. I've learned lot of interesting behavior of the C language. But after several weeks/months of reading, testing, ... I've discovered some weird coding elements. Finally, I've made few changes and I've proposed them to William.
William asked me to assure that those changes resist to a performance benchmark like Apache Benchmark can do. Moreover, he asked me to assure that the changes will not generate memory leaks.
The benchmarking tool
On OpenBSD the ApacheBenchmark is inside the Apache package. To avoid such installation, I've decided to use the "hey" go program: https://github.com/rakyll/hey
To have it on OpenBSD, you just have to do:
pkg_add go
go get -u github.com/rakyll/hey
Then, you will have, in your path: "go/bin/hey".
My changes in Fapws
Since my account is at Sourceforge, I've pushed my commit their (not in github).
The changes are build on my OpenBSD 6.1 machine.
In short, I've changed the way input_header and response_header are managed.
I let you check them in sourceforge -> code -> history
Benchmark results of Fapws
Fapws: a simple hello script
Since I've done benchmarking checks to validate my changes, let me share some of my observations. But before lest me share few topics:
- Please note that, I've taken scripts available in Fapws.
- I've always used the same benchmark parameters.
- Both the webserver and benchmark tool are running the same host (OpenBSD 6.1). This is not representative, but on the other side, since I do not have a decent and nice network, I avoid some network impacts.
Here after the results I've got with Fapws.
sample/hello/hello.py:
./hey -n 10000 -c 100 http://127.0.0.1:8080/hello
Total: 2.1926 secs
Slowest: 0.0734 secs
Fastest: 0.0002 secs
Average: 0.0214 secs
Requests/sec: 4560.8528
Fapws: a simple file
For this test, I'm using the script provide in the test folder of Fapws.
tests/unittests/server.py
./hey -n 10000 -c 100 http://127.0.0.1:8080/short
Total: 4.0994 secs
Slowest: 0.0909 secs
Fastest: 0.0007 secs
Average: 0.0405 secs
Requests/sec: 2439.3703
It's important to note, that all of those tests have returned "200" as return code. Thus none of the requests have been lost, broken, ...
After few iterations, I was happy to see that my changes do not create other crashes or memory leaks.
But before taking any conclusion, I've decided to compare this tool to httpd.
Indeed, Httpd is now present in every OpenBSD install (it's in base). Httpd with cgi could maybe provide a better results ?
How does httpd will perform?
Httpd.conf
First, let me share the config I've used for httpd.
/etc/htttpd.conf
ext_addr="*"
#
# Global Options
#
# prefork 3
prefork 1
#
# Servers
#
# A minimal default server
server "default" {
listen on $ext_addr port 80
location "*.cgi" {
fastcgi
root "/"
}
}
As you can see, to have of better point of comparison, I've configured httpd to have only 1 prefork.
httpd simple file results
As having created a simple index.html with "hello world" in /var/www/htdocs. The results of httpd are:
httpd simple file:
./hey -n 10000 -c 100 http://127.0.0.1/index.html
Total: 1.3358 secs
Slowest: 0.0534 secs
Fastest: 0.0008 secs
Average: 0.0132 secs
Requests/sec: 7486.0689
httpd with a simple cgi script
But we need to benchmark it for dynamic content too.
httpd comes with a tool dedicated to that: slowcgi.
As stated in the httpd.conf file here above, all files ended by ".cgi" will be treated by slowcgi listening on the default socket: /var/www/run/slowcgi.sock
Then, I've added in the chroot environment "/bin/sh" (cp /bin/sh /var/www/bin) and I've created a simple cgi script like this:
#!/bin/sh
echo "Content-type: text/html"
echo ""
echo "<html><body>hello</body></html>"
To assure the file will be correctly executed by httpd, you can check it with the command:
chroot -u www /var/www cgi-bin/test.cgi
In fact httpd will run cgi script inside a chroot environment "/var/www" with the user "www"
If it runs without error, you are ready to got it via httpd and slowcgi.
My benchmark via "hey" provide me the following results;
./hey -n 10000 -c 100 http://127.0.0.1/cgi-bin/test.cgi
Total: 15.4987 secs
Slowest: 0.2497 secs
Fastest: 0.0146 secs
Average: 0.1540 secs
Requests/sec: 645.2152
The results are much less interesting.
So, let's try with a real FastCGI server.
httpd with a FastCGI server
Flup sounds to be the most simple and easy to install FastCGI server.
I've installed it with the command:
pip2.7 install flup
My test script is:
from cgi import escape
import sys, os
from flup.server.fcgi import WSGIServer
def app(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/html')])
yield 'hello world!!'
WSGIServer(app, bindAddress='/var/www/run/slowcgi.sock').run()
Please note that the socket is the same as for slowcgi. Thus I've killed slowcgi, but I have not changed the httpd.conf file.
Thanks to this setup I've got the following benchmark results:
./hey -n 10000 -c 100 http://127.0.0.1/test
Total: 5.9791 secs
Slowest: 3.0016 secs
Fastest: 0.0004 secs
Average: 0.0534 secs
Requests/sec: 1672.4885
This is much better, than pure cgi :-)
Memory consumption
Because, in my case, the webserver is a nice-to-have component; memory allocated to the webserver could help me to perform a final choice between the 2 solutions.
Here after result of a "ps aux" command:
httpd:
root 24002 0.0 0.0 1228 2132 ?? Ssp 4:54PM 0:00.02 /usr/sbin/httpd
www 15870 0.0 0.1 2008 7900 ?? Isp 4:54PM 0:01.40 httpd: server (httpd)
www 65976 0.0 0.1 2016 2548 ?? Isp 4:54PM 0:01.40 httpd: logger (httpd)
www 71791 0.0 0.2 21404 16176 p1 S+ 4:51PM 0:26.90 python2.7 testflup.py
www 8872 6.1 0.0 560 1308 ?? Isp 4:57PM 0:13.34 slowcgi
fapws:
vi 966 0.0 0.2 7096 13064 p0 I+ 4:59PM 0:03.11 python2.7 server.py
We can see that httpd + slowcgi takes same memory as Fapws. But be careful that Fapws will put most of his scripts in memory. Thus more complex is your webserver, more resources you will allocated to your webserver.
I've never run slowcgi on a more complex website, so I cannot say if slowcgi will also take more space in memory.
Nevertheless, we see that flup take more space. As for Fapws, more functionalities you put in it, more memory you'll use.
Conclusion
Finally, after those tests, I'm not able to decide. Fapws is really simple to deploy and to use. But his code is far from the best.
Httpd is remarkable build, included in "base", but we have to associated it a FastCGI server.
I think that I'll have to rewrite all my scripts from WSGI to FastCGI
But then, I still have to find a FastCGI server.
In the end, does a FastCGI server is really different than Fapws ?
Personally, and to not rewrite my web services, I'll stay with Fapws. They are not taking so much space in memory and fapws is very simple to install. Moreover, now , I understand his coding structure.
Nevertheless, if you have lot of static files, httpd is defacto the best solution you have.
Annex
For those who are interested, I've also make measurements with httpd having a prefork = 5.
simple file:
./hey -n 10000 -c 100 http://127.0.0.1/index.html
Total: 1.0025 secs
Slowest: 0.0626 secs
Fastest: 0.0002 secs
Average: 0.0082 secs
Requests/sec: 9974.6884
Total data: 210000 bytes
Size/request: 21 bytes
test.cgi:
./hey -n 10000 -c 100 http://127.0.0.1/cgi-bin/test.cgi
Total: 14.1771 secs
Slowest: 0.2058 secs
Fastest: 0.0304 secs
Average: 0.1410 secs
Requests/sec: 705.3652
flup fcgi:
./hey -n 10000 -c 100 http://127.0.0.1/test
Total: 6.0436 secs
Slowest: 3.1270 secs
Fastest: 0.0004 secs
Average: 0.0588 secs
Requests/sec: 1654.6420