列的数量是否会影响SQLAlchemy的速度？

Question

我创建两个表SQLAlchemy的（蟒2.7），数据库是MySQL的5.5。 以下是我的代码：

engine = create_engine('mysql://root:123@localhost/test') 

metadata = MetaData() 

conn = engin.connect() 

# For table 1: 

columns = [] 

for i in xrange(100): 

    columns.append(Column('c%d' % i, TINYINT, nullable = False, server_default = '0')) 
    columns.append(Column('d%d' % i, SmallInteger, nullable = False, server_default = '0')) 

user = Table('user', metadata, *columns) 
# So user has 100 tinyint columns and 100 smallint columns. 

# For table 2: 

user2 = Table('user2', metadata, 

     Column('c0', BINARY(100), nullable = False, server_default='\0'*100), 
     Column('d0', BINARY(200), nullable = False, server_default='\0'*200), 
) 

# user2 has two columns contains 100 bytes and 200 bytes respectively. 

I then inserted 4000 rows into each table. Since these two tables have same row length, I 
expect the select speed will be almost the same. I ran the following test code: 

s1 = select([user]).compile(engine) 

s2 = select([user2]).compile(engine) 

t1 = time() 

result = conn.execute(s1).fetchall() 

print 't1:', time() - t1 

t2 = time() 

result = conn.execute(s2).fetchall() 

print 't2', time() - t2 

The result is : 

t1: 0.5120000 

t2: 0.0149999 

这是否意味着在表中的列数将显着影响的SQLAlchemy的性能 ？先谢谢你！

Answer 1

这是否意味着表中的列数会显着影响SQLAlchemy的性能？

以及那是一个艰难的一个，它可能更依赖于底层的SQL引擎，MySQL在这种情况下，后来居然sqlalchemy，这只不过是一种方式更同时使用相同的接口，不同的数据库引擎互动。

SQLAlchemy是Python SQL工具包和对象关系映射器，为应用程序开发人员提供了SQL的全部功能和灵活性。

它提供了一套完整的众所周知的企业级持久性模式，实现高效和高性能的数据库访问，改编成简单和Python的领域语言。

虽然我可能是错的，你可以尝试使用普通SQL基准它。

我居然跑了一些测试...

import timeit 

setup = """ 
from sqlalchemy import create_engine, MetaData, select, Table, Column 
from sqlalchemy.dialects.sqlite import BOOLEAN, SMALLINT, VARCHAR 
engine = create_engine('sqlite://', echo = False) 
metadata = MetaData() 
conn = engine.connect() 
columns = [] 

for i in xrange(100): 
    columns.append(Column('c%d' % i, VARCHAR(1), nullable = False, server_default = '0')) 
    columns.append(Column('d%d' % i, VARCHAR(2), nullable = False, server_default = '00')) 


user = Table('user', metadata, *columns) 
user.create(engine) 
conn.execute(user.insert(), [{}] * 4000) 

user2 = Table('user2', metadata, Column('c0', VARCHAR(100), nullable = False, server_default = '0' * 100), \ 
           Column('d0', VARCHAR(200), nullable = False, server_default = '0' * 200)) 
user2.create(engine) 
conn.execute(user2.insert(), [{}] * 4000) 
""" 

many_columns = """ 
s1 = select([user]).compile(engine) 
result = conn.execute(s1).fetchall() 
""" 

two_columns = """ 
s2 = select([user2]).compile(engine) 
result = conn.execute(s2).fetchall() 
""" 

raw_many_columns = "res = conn.execute('SELECT * FROM user').fetchall()" 
raw_two_columns = "res = conn.execute('SELECT * FROM user2').fetchall()" 

timeit.Timer(two_columns, setup).timeit(number = 1) 
timeit.Timer(raw_two_columns, setup).timeit(number = 1) 
timeit.Timer(many_columns, setup).timeit(number = 1) 
timeit.Timer(raw_many_columns, setup).timeit(number = 1) 

>>> timeit.Timer(two_columns, setup).timeit(number = 1) 
0.010751008987426758 
>>> timeit.Timer(raw_two_columns, setup).timeit(number = 1) 
0.0099620819091796875 
>>> timeit.Timer(many_columns, setup).timeit(number = 1) 
0.23563408851623535 
>>> timeit.Timer(raw_many_columns, setup).timeit(number = 1) 
0.21881699562072754 

我没有发现这样的：
http://www.mysqlperformanceblog.com/2009/09/28/how-number-of-columns-affects-performance/

这是一种有趣的，虽然他用max测试...

我真的爱SQLAlchemy的，所以我决定用自己的蟒蛇sqlite3的模块来比较它

import timeit 
setup = """ 
import sqlite3 
conn = sqlite3.connect(':memory:') 
c = conn.cursor() 

c.execute('CREATE TABLE user (%s)' %\ 
      ("".join(("c%i VARCHAR(1) DEFAULT '0' NOT NULL, d%i VARCHAR(2) DEFAULT '00' NOT NULL," % (index, index) for index in xrange(99))) +\ 
      "c99 VARCHAR(1) DEFAULT '0' NOT NULL, d99 VARCHAR(2) DEFAULT '0' NOT NULL")) 

c.execute("CREATE TABLE user2 (c0 VARCHAR(100) DEFAULT '%s' NOT NULL, d0 VARCHAR(200) DEFAULT '%s' NOT NULL)" % ('0'* 100, '0'*200)) 

conn.commit() 
c.executemany('INSERT INTO user VALUES (%s)' % ('?,' * 199 + '?'), [('0',) * 200] * 4000) 
c.executemany('INSERT INTO user2 VALUES (?,?)', [('0'*100, '0'*200)] * 4000) 
conn.commit() 
""" 

many_columns = """ 
r = c.execute('SELECT * FROM user') 
all = r.fetchall() 
""" 

two_columns = """ 
r2 = c.execute('SELECT * FROM user2') 
all = r2.fetchall() 
""" 

timeit.Timer(many_columns, setup).timeit(number = 1) 
timeit.Timer(two_columns, setup).timeit(number = 1) 

>>> timeit.Timer(many_columns, setup).timeit(number = 1) 
0.21009302139282227 
>>> timeit.Timer(two_columns, setup).timeit(number = 1) 
0.0083379745483398438 

，并想出了相同的结果，所以我真的不认为这是一个数据库实现不是sqlalchemy问题。

DEFAULT INSERT

import timeit 

setup = """ 
from sqlalchemy import create_engine, MetaData, select, Table, Column 
from sqlalchemy.dialects.sqlite import BOOLEAN, SMALLINT, VARCHAR 
engine = create_engine('sqlite://', echo = False) 
metadata = MetaData() 
conn = engine.connect() 
columns = [] 

for i in xrange(100): 
    columns.append(Column('c%d' % i, VARCHAR(1), nullable = False, server_default = '0')) 
    columns.append(Column('d%d' % i, VARCHAR(2), nullable = False, server_default = '00')) 


user = Table('user', metadata, *columns) 
user.create(engine) 

user2 = Table('user2', metadata, Column('c0', VARCHAR(100), nullable = False, server_default = '0' * 100), \ 
           Column('d0', VARCHAR(200), nullable = False, server_default = '0' * 200)) 
user2.create(engine) 
""" 

many_columns = """ 
conn.execute(user.insert(), [{}] * 4000) 
""" 

two_columns = """ 
conn.execute(user2.insert(), [{}] * 4000) 
""" 

>>> timeit.Timer(two_columns, setup).timeit(number = 1) 
0.017949104309082031 
>>> timeit.Timer(many_columns, setup).timeit(number = 1) 
0.047809123992919922 

测试与sqlite3的模块。

import timeit 
setup = """ 
import sqlite3 
conn = sqlite3.connect(':memory:') 
c = conn.cursor() 

c.execute('CREATE TABLE user (%s)' %\ 
    ("".join(("c%i VARCHAR(1) DEFAULT '0' NOT NULL, d%i VARCHAR(2) DEFAULT '00' NOT NULL," % (index, index) for index in xrange(99))) +\ 
      "c99 VARCHAR(1) DEFAULT '0' NOT NULL, d99 VARCHAR(2) DEFAULT '0' NOT NULL")) 

c.execute("CREATE TABLE user2 (c0 VARCHAR(100) DEFAULT '%s' NOT NULL, d0 VARCHAR(200) DEFAULT '%s' NOT NULL)" % ('0'* 100, '0'*200)) 
conn.commit() 
""" 

many_columns = """ 
c.executemany('INSERT INTO user VALUES (%s)' % ('?,' * 199 + '?'), [('0', '00') * 100] * 4000) 
conn.commit() 
""" 

two_columns = """ 
c.executemany('INSERT INTO user2 VALUES (?,?)', [('0'*100, '0'*200)] * 4000) 
conn.commit() 
""" 

timeit.Timer(many_columns, setup).timeit(number = 1) 
timeit.Timer(two_columns, setup).timeit(number = 1) 

>>> timeit.Timer(many_columns, setup).timeit(number = 1) 
0.14044189453125 
>>> timeit.Timer(two_columns, setup).timeit(number = 1) 
0.014360189437866211 
>>>  

Answer 2

Samy.vilar的回答非常好。但需要记住的一点是，列数会影响任何数据库和任何ORM的性能。您拥有的列越多，从磁盘访问和传输的数据就越多。

此外，根据查询和表结构，增加更多的列可以被覆盖的索引来被迫访问基台，其可以在一定的数据库和某些情况下添加大量的时间而改变的查询。

我只用了一点SQLAlchemy，但作为一名DBA，我通常会建议我所使用的开发人员仅查询他们需要的列，并避免在生产代码中使用“select *”，这两者都是因为它可能会包含比所需更多的列，因为它会使代码在添加到表/视图中的潜在列的表面更加脆弱。