博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
psycopg2 postgresql driver for python don't support prepared statement Direct
阅读量:6412 次
发布时间:2019-06-23

本文共 6720 字,大约阅读时间需要 22 分钟。

前面使用py-postgresql测试过PostgreSQL性能, 可能是这个驱动效率较低, 我们接下来使用psycopg2测试一下.
psycopg2使用libpq接口, 支持2PC, 支持异步提交等, 
但是不支持绑定变量.
安装
[root@localhost ~]# . /home/postgres/.bash_profile root@localhost-> which pg_config/opt/pgsql9.3.5/bin/pg_config[root@localhost ~]# pip3.4 install psycopg2 --upgrade或[root@localhost ~]# pip3.4 install psycopg2
异步操作, 对数据库端来说意义不大, 因为在执行的话, 不能再次提交SQL请求.
如下 :
import psycopg2import timeimport selectconn = psycopg2.connect(database="postgres", user="postgres", password="postgres", host="/data01/pgdata/pg_root", port="1921", async=True)>>> def wait(conn):...     while 1:...         state = conn.poll()...         if state == psycopg2.extensions.POLL_OK:...             break...         elif state == psycopg2.extensions.POLL_WRITE:...             select.select([], [conn.fileno()], [])...         elif state == psycopg2.extensions.POLL_READ:...             select.select([conn.fileno()], [], [])...         else:...             raise psycopg2.OperationalError("poll() returned %s" % state)... wait(conn)curs = conn.cursor()curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": 1})curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": 1})Traceback (most recent call last):  File "
", line 1, in
psycopg2.ProgrammingError: execute cannot be used while an asynchronous query is underway>>> wait(curs.connection)>>> wait(curs.connection)>>> curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": 1})>>> wait(curs.connection)>>> curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": 1})>>> wait(curs.connection)>>> conn.poll()0>>> curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": 1})>>> conn.poll()0>>> curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": 1})>>> conn.poll()0>>> psycopg2.extensions.POLL_OK0>>> psycopg2.extensions.POLL_WRITE2>>> psycopg2.extensions.POLL_READ1
同时psycopg2不支持prepared statement
>>> curs.prepare("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')")Traceback (most recent call last):  File "
", line 1, in
AttributeError: 'psycopg2._psycopg.cursor' object has no attribute 'prepare'>>> dir(curs)['__class__', '__delattr__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'arraysize', 'binary_types', 'callproc', 'cast', 'close', 'closed', 'connection', 'copy_expert', 'copy_from', 'copy_to', 'description', 'execute', 'executemany', 'fetchall', 'fetchmany', 'fetchone', 'itersize', 'lastrowid', 'mogrify', 'name', 'nextset', 'query', 'row_factory', 'rowcount', 'rownumber', 'scroll', 'scrollable', 'setinputsizes', 'setoutputsize', 'statusmessage', 'string_types', 'typecaster', 'tzinfo_factory', 'withhold']
为什么这么说呢?, 调用
curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": 1})
你觉得是绑定变量吗? 看看PostgreSQL的日志吧 :
2015-02-05 22:37:32.901 CST,"postgres","postgres",69785,"[local]",54d37ee8.11099,1,"idle",2015-02-05 22:32:08 CST,3/1996971,0,LOG,00000,"statement: insert into tt values(1, 'digoal.zhou', 32, 'digoal@126.com', '276732431')",,,,,,,,"exec_simple_query, postgres.c:890",""
调用的是exec_simple_query接口, 当然不是绑定变量.

间接的使用绑定变量, 但实际上SQL调用的还是
exec_simple_query接口, 只是在处理execute这个SQL时使用到了绑定变量.
这个是较老的用法.
>>> curs.execute("prepare pre1(int) as insert into tt values($1, 'digoal.zhou', 32, 'digoal@126.com', '276732431')")>>> conn.poll()0>>> curs.execute("execute pre1(1)")>>> conn.poll()0>>> curs.execute("execute pre1(1)")>>> conn.poll()0
PostgreSQL日志 :
2015-02-05 22:50:42.678 CST,"postgres","postgres",69785,"[local]",54d37ee8.11099,5,"idle",2015-02-05 22:32:08 CST,3/1996975,0,LOG,00000,"statement: prepare pre1(int) as insert into tt values($1, 'digoal.zhou', 32, 'digoal@126.com', '276732431')",,,,,,,,"exec_simple_query, postgres.c:890",""2015-02-05 22:50:59.425 CST,"postgres","postgres",69785,"[local]",54d37ee8.11099,6,"idle",2015-02-05 22:32:08 CST,3/1996976,0,LOG,00000,"statement: execute pre1(1)","prepare: prepare pre1(int) as insert into tt values($1, 'digoal.zhou', 32, 'digoal@126.com', '276732431')",,,,,,,"exec_simple_query, postgres.c:890",""2015-02-05 22:51:22.425 CST,"postgres","postgres",69785,"[local]",54d37ee8.11099,7,"idle",2015-02-05 22:32:08 CST,3/1996977,0,LOG,00000,"statement: execute pre1(1)","prepare: prepare pre1(int) as insert into tt values($1, 'digoal.zhou', 32, 'digoal@126.com', '276732431')",,,,,,,"exec_simple_query, postgres.c:890",""
效率如何呢?
使用8个线程插入100W数据, 耗时41秒.
py-postgresql驱动耗时226秒, psycopg2效率高了很多, 接近pgbench的16秒了.
[root@localhost ~]# cat t.pyimport psycopg2import timeimport threadingclass n_t(threading.Thread):   #The timer class is derived from the class threading.Thread  def __init__(self, num):    threading.Thread.__init__(self)    self.thread_num = num  def run(self): #Overwrite run() method, put what you want the thread do here    conn = psycopg2.connect(database="postgres", user="postgres", password="postgres", host="/data01/pgdata/pg_root", port="1921")    curs = conn.cursor()    conn.autocommit=True    start_t = time.time()    print("TID:" + str(self.thread_num) + " " + str(start_t))    for i in range(self.thread_num*125000, (self.thread_num+1)*125000):      curs.execute("insert into tt values(%(id)s, 'digoal.zhou', 32, 'digoal@126.com', '276732431')", {"id": i})    stop_t = time.time()    print("TID:" + str(self.thread_num) + " " + str(stop_t))    print(stop_t-start_t)def test():  t_names = dict()  for i in range(0,8):    t_names[i] = n_t(i)     t_names[i].start()  returnif __name__ == '__main__':  test()
测试结果 : 
[root@localhost ~]# python t.pyTID:0 1423148404.5820847TID:1 1423148404.5828164TID:2 1423148404.5850544TID:3 1423148404.58532TID:4 1423148404.5861893TID:5 1423148404.5867805TID:6 1423148404.5882406TID:7 1423148404.588671TID:2 1423148445.14644940.561394691467285TID:7 1423148445.205186140.616515159606934TID:6 1423148445.232101240.64386057853699TID:1 1423148445.26323640.68041968345642TID:5 1423148445.270324240.68354368209839TID:0 1423148445.296777540.71469283103943TID:3 1423148445.306561740.72124171257019TID:4 1423148445.334591640.74840235710144postgres=# select count(*),count(distinct id) from tt;  count  |  count  ---------+--------- 1000000 | 1000000(1 row)
[参考]
1. 
2. 

3. 

转载地址:http://mwura.baihongyu.com/

你可能感兴趣的文章
[转]Reporting Service部署之访问权限
查看>>
innerxml and outerxml
查看>>
validform校验框架不显示错误提示
查看>>
flink 获取上传的Jar源码
查看>>
Spring Data JPA Batch Insertion
查看>>
UEditor自动调节宽度
查看>>
JAVA做验证码图片(转自CSDN)
查看>>
Delphi TServerSocket,TClientSocket实现传送文件代码
查看>>
JS无聊之作
查看>>
Mac上搭建ELK
查看>>
443 Chapter7.Planning for High Availability in the Enterprise
查看>>
框架和语言的作用
查看>>
unidac连接ORACLE免装客户端驱动
查看>>
Cygwin + OpenSSH FOR Windows的安装配置
查看>>
咏南中间件支持手机客户端
查看>>
fastscript增加三方控件之二
查看>>
Windows Vista RTM 你准备好了么?
查看>>
Tensorflow Serving 模型部署和服务
查看>>
Java Web开发详解——XML+DTD+XML Schema+XSLT+Servlet 3.0+JSP 2.2深入剖析与实例应用
查看>>
topcoder srm 680 div1 -3
查看>>