RELATEED CONSULTING
相关咨询
选择下列产品马上在线沟通
服务时间：8:30-17:00
你可能遇到了下面的问题
关闭右侧工具栏

新闻中心

这里有您想知道的互联网营销解决方案
Python3快速入门（十五）——Pandas数据处理

Python3快速入门（十五）——Pandas数据处理

一、函数应用

1、函数应用简介

如果要将自定义函数或其它库函数应用于Pandas对象，有三种使用方式。pipe()将函数用于表格，apply()将函数用于行或列，applymap()将函数用于元素。

创新互联建站-专业网站定制、快速模板网站建设、高性价比单县网站开发、企业建站全套包干低至880元,成熟完善的模板库,直接使用。一站式单县网站制作公司更省心,省钱,快速模板网站建设找我们，业务覆盖单县地区。费用合理售后完善，10年实体公司更值得信赖。

2、表格函数应用

可以通过将函数对象和参数作为pipe函数的参数来执行自定义操作，会对整个DataFrame执行操作。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

def adder(x, y):
    return x + y

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(5, 3),columns=['col1', 'col2', 'col3'])
    print(df)
    df = df.pipe(adder, 1)
    print(df)

# output:
#        col1      col2      col3
# 0  0.390803  0.940306 -1.300635
# 1 -0.349588 -1.290132  0.415693
# 2 -0.079585 -0.083825  0.262867
# 3  0.582377  0.171701 -1.011748
# 4 -0.466655  1.746269  1.281538
#        col1      col2      col3
# 0  1.390803  1.940306 -0.300635
# 1  0.650412 -0.290132  1.415693
# 2  0.920415  0.916175  1.262867
# 3  1.582377  1.171701 -0.011748
# 4  0.533345  2.746269  2.281538

3、行、列函数应用

使用apply()函数可以沿DataFrame或Panel的轴执行应用函数，采用可选axis参数。 默认情况下，操作按列执行。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

def adder(x, y):
    return x + y

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(5, 3), columns=['col1', 'col2', 'col3'])
    print(df)
    # 按列执行
    result = df.apply(np.sum)
    print(result)
    # 按行执行
    result = df.apply(np.sum, axis=1)
    print(result)

# output:
#        col1      col2      col3
# 0 -1.773775 -0.608478  0.602059
# 1 -0.208412  0.969435 -0.292108
# 2  0.776864 -0.768559 -0.389092
# 3 -2.088412  1.133090  1.006486
# 4  0.693241  1.808845  0.772191
# col1   -2.600494
# col2    2.534332
# col3    1.699536
# dtype: float64
# 0   -1.780194
# 1    0.468915
# 2   -0.380788
# 3    0.051164
# 4    3.274277
# dtype: float64

4、元素函数应用

在DataFrame的applymap()函数可以接受任何Python函数，并且返回单个值。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(5, 3), columns=['col1', 'col2', 'col3'])
    print(df)
    df = df.applymap(lambda x: x + 1)
    print(df)

# output:
#        col1      col2      col3
# 0  2.396185 -0.263581 -0.090799
# 1  1.718716  0.876074 -1.067746
# 2 -1.033945 -0.078448  1.036566
# 3  0.553849  0.251312 -0.422640
# 4 -0.896062  1.605349 -0.089430
#        col1      col2      col3
# 0  3.396185  0.736419  0.909201
# 1  2.718716  1.876074 -0.067746
# 2 -0.033945  0.921552  2.036566
# 3  1.553849  1.251312  0.577360
# 4  0.103938  2.605349  0.910570

二、数据清洗

1、数据清洗简介

数据清洗是一项复杂且繁琐的工作，同时也是数据分析过程中最为重要的环节。数据清洗的目的一是通过清洗让数据可用，二是让数据变的更适合进行数据分析工作。因此，脏数据要清洗，干净数据也要清洗。在实际数据分析中，数据清洗将占用项目70%左右的时间。

2、缺失值处理

查看每一列有多少缺失值。
df.isnull().sum()
查看每一列有多少完整的数据
df.shape[0]-df.isnull().sum()

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 3), index=dates, columns=list('ABC'))
    print(df)
    print(df.isnull().sum())
    print(df.shape[0] - df.isnull().sum())

# output:
#                    A         B         C
# 2019-01-01  1.138325  0.981597  1.359580
# 2019-01-02 -1.622074  0.812393 -0.946351
# 2019-01-03  0.049815  1.194241  0.807209
# 2019-01-04  1.500074 -0.570367 -0.328529
# 2019-01-05  0.465869  1.049651 -0.112453
# 2019-01-06 -1.399495  0.492769  1.961198
# A    0
# B    0
# C    0
# dtype: int64
# A    6
# B    6
# C    6
# dtype: int64

删除列

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    print(df)
    del df['D']
    # 删除第2列
    df.drop(df.columns[2], axis=1, inplace=True)
    # 删除B列
    df.drop('B', axis=1, inplace=True)
    print(df)

# output:
#                    A         B         C
# 2019-01-01 -0.703151  0.753482 -0.624376
# 2019-01-02 -0.396221 -0.832279 -1.419897
# 2019-01-03 -0.179341 -0.368501 -0.300810
# 2019-01-04  0.464156  0.117461  1.502114
# 2019-01-05 -1.022012 -1.612456  1.611377
# 2019-01-06 -0.677521  0.001020 -0.342290
#                    A
# 2019-01-01 -0.703151
# 2019-01-02 -0.396221
# 2019-01-03 -0.179341
# 2019-01-04  0.464156
# 2019-01-05 -1.022012
# 2019-01-06 -0.677521

删除NaN值

df.dropna(self, axis=0, how='any', thresh=None, subset=None,
           inplace=False)

axis为轴，0表示对行进行操作，1表示对列进行操作。
how为操作类型，’any’表示只要出现NaN的行或列都删除，’all’表示删除整行或整列都为NaN的行或列。
thresh：NaN的阈值，达到thresh时删除。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    df.iloc[1, 3] = None
    df.iloc[2, 2] = None
    print(df)
    print(df.dropna(axis=1))
    print(df.dropna(how='any'))

# output:
#                    A         B         C         D
# 2019-01-01 -0.152239 -2.315100 -0.504998 -0.987549
# 2019-01-02 -1.884801  1.046506 -1.618871       NaN
# 2019-01-03  0.976682 -1.043107       NaN  0.391338
# 2019-01-04  0.143389  0.951518  0.040632 -0.443944
# 2019-01-05  3.092766  0.787921 -2.408260 -1.111238
# 2019-01-06 -0.179249  0.573734 -0.912023  0.261517
#                    A         B
# 2019-01-01 -0.152239 -2.315100
# 2019-01-02 -1.884801  1.046506
# 2019-01-03  0.976682 -1.043107
# 2019-01-04  0.143389  0.951518
# 2019-01-05  3.092766  0.787921
# 2019-01-06 -0.179249  0.573734
#                    A         B         C         D
# 2019-01-01 -0.152239 -2.315100 -0.504998 -0.987549
# 2019-01-04  0.143389  0.951518  0.040632 -0.443944
# 2019-01-05  3.092766  0.787921 -2.408260 -1.111238
# 2019-01-06 -0.179249  0.573734 -0.912023  0.261517

填充NaN值

df.fillna(self, value=None, method=None, axis=None, inplace=False,limit=None, downcast=None, **kwargs)

value:填充的值，可以为字典，字典的key为列名称。
inplace:表示是否对源数据进行修改，默认为False。
fillna默认会返回新对象，但也可以对现有对象进行就地修改。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    df.iloc[1, 3] = None
    df.iloc[2, 2] = None
    print(df)
    print(df.fillna({'C': 3.14, 'D': 0.0}))
    # 使用指定值填充
    df.fillna(value=3.14, inplace=True)
    print(df)

# output:
#                    A         B         C         D
# 2019-01-01  0.490727 -0.603079  0.202922  2.012060
# 2019-01-02 -0.855106  0.305557  0.851141       NaN
# 2019-01-03 -0.324215  0.629637       NaN -0.174930
# 2019-01-04  0.085996  0.173265  0.416938 -0.903989
# 2019-01-05  0.009368  0.410056 -1.297822 -2.202893
# 2019-01-06  0.021892 -0.359749 -0.608556 -0.859454
#                    A         B         C         D
# 2019-01-01  0.490727 -0.603079  0.202922  2.012060
# 2019-01-02 -0.855106  0.305557  0.851141  0.000000
# 2019-01-03 -0.324215  0.629637  3.140000 -0.174930
# 2019-01-04  0.085996  0.173265  0.416938 -0.903989
# 2019-01-05  0.009368  0.410056 -1.297822 -2.202893
# 2019-01-06  0.021892 -0.359749 -0.608556 -0.859454
#                    A         B         C         D
# 2019-01-01  0.490727 -0.603079  0.202922  2.012060
# 2019-01-02 -0.855106  0.305557  0.851141  3.140000
# 2019-01-03 -0.324215  0.629637  3.140000 -0.174930
# 2019-01-04  0.085996  0.173265  0.416938 -0.903989
# 2019-01-05  0.009368  0.410056 -1.297822 -2.202893
# 2019-01-06  0.021892 -0.359749 -0.608556 -0.859454

对数据进行布尔填充

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    df.iloc[1, 3] = None
    df.iloc[2, 2] = None
    print(df)
    print(pd.isnull(df))

# output:
#                    A         B         C         D
# 2019-01-01 -1.337471  0.154446  0.493862  1.278946
# 2019-01-02  2.853301 -0.151376  0.318281       NaN
# 2019-01-03  1.094465  0.059063       NaN  0.216805
# 2019-01-04 -0.983091 -1.052905  0.416604 -1.431156
# 2019-01-05 -1.421142  1.015465 -1.851315 -0.680514
# 2019-01-06  0.224378 -0.636699 -0.749040 -0.728368
#                 A      B      C      D
# 2019-01-01  False  False  False  False
# 2019-01-02  False  False  False   True
# 2019-01-03  False  False   True  False
# 2019-01-04  False  False  False  False
# 2019-01-05  False  False  False  False
# 2019-01-06  False  False  False  False

3、行和列处理

通过字典键可以进行列选择，获取DataFrame中的一列数据。
生成DataFrame时指定index和columns

import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    print(df)

# output:
#                    A         B         C         D
# 2013-01-01  1.116914 -0.221035 -0.577299 -0.328831
# 2013-01-02  1.764656  1.462838 -0.360678  1.176134
# 2013-01-03  0.144396 -0.594359 -0.548543  1.281829
# 2013-01-04  0.632378  0.895123 -0.757924 -1.325917
# 2013-01-05  0.219125 -1.247446  0.335363 -0.676052
# 2013-01-06  0.963715 -0.131331  0.326482 -0.718461

index和columns也可以在DataFrame创建后指定

import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    print(df)
    df.index = pd.date_range('20130201', periods=df.shape[0])
    df.columns = list('abcd')
    print(df)
    df.index = pd.date_range('20130301', periods=len(df))
    df.columns = list('ABCD')
    print(df)

# output:
#                    A         B         C         D
# 2013-01-01  1.588442  1.548420  0.132539  0.410512
# 2013-01-02  0.200415  1.515354  2.275575 -1.533603
# 2013-01-03  0.838294  0.067409 -1.157181  0.401973
# 2013-01-04  0.551363 -0.749296  0.343762 -1.558969
# 2013-01-05 -0.799507 -1.343379 -0.006312  1.091014
# 2013-01-06  0.012188 -0.382384  0.280008 -2.333430
#                    a         b         c         d
# 2013-02-01  1.588442  1.548420  0.132539  0.410512
# 2013-02-02  0.200415  1.515354  2.275575 -1.533603
# 2013-02-03  0.838294  0.067409 -1.157181  0.401973
# 2013-02-04  0.551363 -0.749296  0.343762 -1.558969
# 2013-02-05 -0.799507 -1.343379 -0.006312  1.091014
# 2013-02-06  0.012188 -0.382384  0.280008 -2.333430
#                    A         B         C         D
# 2013-03-01  1.588442  1.548420  0.132539  0.410512
# 2013-03-02  0.200415  1.515354  2.275575 -1.533603
# 2013-03-03  0.838294  0.067409 -1.157181  0.401973
# 2013-03-04  0.551363 -0.749296  0.343762 -1.558969
# 2013-03-05 -0.799507 -1.343379 -0.006312  1.091014
# 2013-03-06  0.012188 -0.382384  0.280008 -2.333430

可以指定某一列为index

import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
    df['date'] = dates
    print(df)
    df = df.set_index('date', drop=True)
    print(df)

# output:
#           A         B         C         D       date
# 0  0.910416 -0.378195  0.332562 -0.194766 2013-01-01
# 1  0.533733  0.888629 -0.358143  1.583278 2013-01-02
# 2  0.482362 -0.905558  1.045753 -0.874653 2013-01-03
# 3  0.901622 -0.535862 -0.439763 -0.640594 2013-01-04
# 4 -1.273577 -0.746785  1.448309 -0.368285 2013-01-05
# 5  0.191289 -1.246213  0.184757 -1.143074 2013-01-06
#                    A         B         C         D
# date
# 2013-01-01  0.910416 -0.378195  0.332562 -0.194766
# 2013-01-02  0.533733  0.888629 -0.358143  1.583278
# 2013-01-03  0.482362 -0.905558  1.045753 -0.874653
# 2013-01-04  0.901622 -0.535862 -0.439763 -0.640594
# 2013-01-05 -1.273577 -0.746785  1.448309 -0.368285
# 2013-01-06  0.191289 -1.246213  0.184757 -1.143074

在原有DataFrame的基础上，可以创建一个新的DataFrame，或者将原有DataFrame按行进行汇总统计创建一个新的DataFrame。

import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 3), index=dates, columns=list('ABC'))
    print(df)
    df1 = pd.DataFrame()
    df1['min'] = df.min()
    df1['max'] = df.max()
    df1['std'] = df.std()
    print(df1)
    df['min'] = df.min(axis=1)
    df['max'] = df.max(axis=1)
    df['std'] = df.std(axis=1)
    print(df)

# output:
#                    A         B         C
# 2013-01-01  0.901073  1.706925 -0.503194
# 2013-01-02  0.379870  0.729674  0.579337
# 2013-01-03 -1.285323 -0.665951 -0.161148
# 2013-01-04 -0.714282  0.423376  0.586061
# 2013-01-05 -0.895171 -0.413328  0.485803
# 2013-01-06  1.926472 -0.718467  1.113522
#         min       max       std
# A -1.285323  1.926472  1.234084
# B -0.718467  1.706925  0.955797
# C -0.503194  1.113522  0.582913
#                    A         B         C       min       max       std
# 2013-01-01  0.901073  1.706925 -0.503194 -0.503194  1.706925  1.113132
# 2013-01-02  0.379870  0.729674  0.579337  0.379870  0.729674  0.175247
# 2013-01-03 -1.285323 -0.665951 -0.161148 -1.285323 -0.161148  0.562671
# 2013-01-04 -0.714282  0.423376  0.586061 -0.714282  0.586061  0.685749
# 2013-01-05 -0.895171 -0.413328  0.485803 -0.895171  0.485803  0.696763
# 2013-01-06  1.926472 -0.718467  1.113522 -0.718467  1.926472  1.341957

axis=0，对DataFrame的每一列数据进行统计运算，得到一行。axis=0，对DataFrame的每一行数据进行统计运算，得到一列。
DataFrame可以修改index和columns。

import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 3), index=dates, columns=list('ABC'))
    print(df)
    df = df.rename(index=lambda x: x + 5, columns={'A': 'newA', 'B': 'newB'})
    print(df)

# output:
#                   A         B         C
# 2013-01-01  0.834910  0.652175  0.537611
# 2013-01-02  1.083902  0.836208 -1.466876
# 2013-01-03 -0.044256  0.932547  1.843682
# 2013-01-04  1.610113 -0.705734 -0.145042
# 2013-01-05  1.114897  0.273569 -0.047725
# 2013-01-06 -0.541942 -0.112752  1.644338
#                 newA      newB         C
# 2013-01-06  0.834910  0.652175  0.537611
# 2013-01-07  1.083902  0.836208 -1.466876
# 2013-01-08 -0.044256  0.932547  1.843682
# 2013-01-09  1.610113 -0.705734 -0.145042
# 2013-01-10  1.114897  0.273569 -0.047725
# 2013-01-11 -0.541942 -0.112752  1.644338

列数据的单位统一

import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    df['D'] = [10000, 34000, 60000, 34000, 56000, 80000]
    print(df)
    for i in range(len(df['D'])):
        weight = float(df.iloc[i, 3]) / 10000
        df.iloc[i, 3] = '{}万'.format(weight)
    print(df)

# output:
#                    A         B         C      D
# 2019-01-01 -0.889533 -0.411451  0.563969  10000
# 2019-01-02 -0.573239  0.264805 -0.058530  34000
# 2019-01-03  1.224993 -1.815338 -2.075301  60000
# 2019-01-04  0.266483  1.841926 -0.759681  34000
# 2019-01-05 -0.167595  0.432617  0.533577  56000
# 2019-01-06 -0.973877  0.700821  1.093101  80000
#                    A         B         C     D
# 2019-01-01 -0.889533 -0.411451  0.563969  1.0万
# 2019-01-02 -0.573239  0.264805 -0.058530  3.4万
# 2019-01-03  1.224993 -1.815338 -2.075301  6.0万
# 2019-01-04  0.266483  1.841926 -0.759681  3.4万
# 2019-01-05 -0.167595  0.432617  0.533577  5.6万
# 2019-01-06 -0.973877  0.700821  1.093101  8.0万

4、重复值删除

df.duplicated(self, subset=None, keep='first')
检查DataFrame是否有重复数据。
subset：子集，列标签或列标签的序列
keep：可选值为first，last，False，first表示保留第一个出现的值，last表示保留最后一个出现的值，False表示保留所有的值。
df.drop_duplicates(self, subset=None, keep='first', inplace=False)
删除DataFrame的重复数据。
subset：子集，列标签或列标签的序列
keep：可选值为first，last，False，first表示保留第一个出现的值，last表示保留最后一个出现的值，False表示保留所有的值。
inplace：值为True表示修改源数据，值为False表示不修改源数据

import pandas as pd
import numpy as np

if __name__ == "__main__":
    data = [['Alex', np.nan, 80], ['Bob', 25, 90], ['Bob', 25, 90]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    # 使用bool过滤，取出重复的值
     print(df[df.duplicated(keep=False)])
    # 删除重复值，修改源数据
     df.drop_duplicates(keep='last', inplace=True)
    print(df)

# output:
#    Name   Age  Score
# 0  Alex   NaN     80
# 1   Bob  25.0     90
# 2   Bob  25.0     90
#   Name   Age  Score
# 1  Bob  25.0     90
# 2  Bob  25.0     90
#    Name   Age  Score
# 0  Alex   NaN     80
# 2   Bob  25.0     90

5、异常值处理

异常值分为两种，一种是非法数据，如数字列的中间夹杂着一些汉字或者是符号；第二种是异常数据，异乎寻常的大数值或者是小数值。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

def swap(x):
    if type(x) == str:
        if x[-1] == '岁':
            x = int(x[:-1])
        elif x[-1] == '分':
            x = int(x[:-1])
    return x

if __name__ == "__main__":
    data = [['Alex', np.nan, '89分'], ['Bob', '25岁', '90分'], ['Bob', '28岁', '90分']]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    df = df.applymap(swap)
    print(df)

# output:
#    Name  Age Score
# 0  Alex  NaN   89分
# 1   Bob  25岁   90分
# 2   Bob  28岁   90分
#    Name   Age  Score
# 0  Alex   NaN     89
# 1   Bob  25.0     90
# 2   Bob  28.0     90

6、数据格式清洗

清除字段字符的前后空格
df[‘city’]=df[‘city’].map(str.strip)
将字段进行大小写转换：
df[‘city’]=df[‘city’].str.lower()

import pandas as pd
import numpy as np

if __name__ == "__main__":
    data = [['Alex', np.nan, 80], [' Bob ', 25, 90], [' Bob', 25, 90]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    # 清除字符串前后空格
    print(df['Name'].map(str.strip))
    # 大小写转换
    print(df['Name'].str.lower())

# output:
#     Name   Age  Score
# 0   Alex   NaN     80
# 1   Bob   25.0     90
# 2    Bob  25.0     90
# 0    Alex
# 1     Bob
# 2     Bob
# Name: Name, dtype: object
# 0     alex
# 1     bob 
# 2      bob
# Name: Name, dtype: object

更改列的数据类型：
df[‘price’].astype(‘int’)

7、数据替换

df[‘city’].replace(‘sh’, ‘shanghai’)
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data = [['Alex', np.nan, 80], ['Bob', 25, 90], ['Bob', 25, 90]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df['Name'].replace('Bob', 'Bauer'))

# output:
#    Name   Age  Score
# 0  Alex   NaN     80
# 1   Bob  25.0     90
# 2   Bob  25.0     90
# 0     Alex
# 1    Bauer
# 2    Bauer
# Name: Name, dtype: object

替换时，字符串前后不能有空格存在，必须严格匹配。

三、数据处理

1、排序

（1）按标签排序

sort_index(self, axis=0, level=None, ascending=True, inplace=False,
               kind='quicksort', na_position='last', sort_remaining=True,
               by=None)

使用sort_index()函数，通过传递axis参数和排序顺序，可以对DataFrame进行排序。 默认情况下，按照升序对行标签进行排序。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(4, 3), index=['rank2', 'rank1', 'rank4', 'rank3'], columns=['col1', 'col2', 'col3'])
    print(df)
    df = df.sort_index()
    print(df)

# output:
#            col1      col2      col3
# rank2 -0.627700 -0.361006 -1.126366
# rank1 -1.997538  1.569461  0.454773
# rank4 -0.598688  1.348594  0.777791
# rank3 -0.190794 -1.209312  0.830699
#            col1      col2      col3
# rank1 -1.997538  1.569461  0.454773
# rank2 -0.627700 -0.361006 -1.126366
# rank3 -0.190794 -1.209312  0.830699
# rank4 -0.598688  1.348594  0.777791

通过将布尔值传递给升序参数ascending，可以控制排序顺序；通过传递axis参数值为1，可以对列标签进行排序。 默认情况下，axis = 0，对行标签进行排序。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(4, 3), index=['rank2', 'rank1', 'rank4', 'rank3'], columns=['col3', 'col2', 'col1'])
    print(df)
    # 按列标签进行排序
    df = df.sort_index(ascending=True, axis=1)
    print(df)

# output:
#            col3      col2      col1
# rank2 -0.715319 -0.245760 -1.282737
# rank1  0.046705 -0.202133  0.185576
# rank4 -1.608270 -0.491281  0.047686
# rank3 -1.013456 -0.020197  1.184151
#            col1      col2      col3
# rank2 -1.282737 -0.245760 -0.715319
# rank1  0.185576 -0.202133  0.046705
# rank4  0.047686 -0.491281 -1.608270
# rank3  1.184151 -0.020197 -1.013456

（2）按值排序

sort_values(self, by, axis=0, ascending=True, inplace=False,
                kind='quicksort', na_position='last')

使用sort_values函数可以按值排序，接收一个by参数，使用DataFrame的列名称作为值，根据某列进行排序。by可以是列名称的列表。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(4, 3), index=['rank2', 'rank1', 'rank4', 'rank3'], columns=['col3', 'col2', 'col1'])
    print(df)
    df = df.sort_values(by="col2")
    print(df)

# output:
#            col3      col2      col1
# rank2 -0.706054 -2.135880  1.066836
# rank1  0.290660 -2.214451 -1.724394
# rank4  1.211874  0.475177 -0.711855
# rank3 -0.253331  1.211301 -0.208633
#            col3      col2      col1
# rank1  0.290660 -2.214451 -1.724394
# rank2 -0.706054 -2.135880  1.066836
# rank4  1.211874  0.475177 -0.711855
# rank3 -0.253331  1.211301 -0.208633

sort_values()提供mergesort，heapsort和quicksort三种排序算法，mergesort是唯一的稳定排序算法，通过参数kind进行传递。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(4, 3), index=['rank2', 'rank1', 'rank4', 'rank3'], columns=['col3', 'col2', 'col1'])
    print(df)
    df = df.sort_values(by="col2", kind='mergesort')
    print(df)

# output:
#            col3      col2      col1
# rank2 -0.243768 -0.344846  0.535481
# rank1 -1.491950  0.690749 -2.023808
# rank4 -0.656292 -0.704788  0.655129
# rank3  0.468007 -0.250702  0.079670
#            col3      col2      col1
# rank4 -0.656292 -0.704788  0.655129
# rank2 -0.243768 -0.344846  0.535481
# rank3  0.468007 -0.250702  0.079670
# rank1 -1.491950  0.690749 -2.023808

按顺序进行多列降序排序

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(4, 3), index=['rank2', 'rank1', 'rank4', 'rank3'], columns=['col3', 'col2', 'col1'])
    print(df)
    df = df.sort_values(by=['col1', 'col3'], ascending=True, axis=0)
    print(df)

# output:
#            col3      col2      col1
# rank2  1.035965  1.048124 -0.341586
# rank1  2.391899 -1.575462  0.616940
# rank4  0.968523 -0.932288 -0.553498
# rank3  0.585521  1.907344 -0.264500
#            col3      col2      col1
# rank4  0.968523 -0.932288 -0.553498
# rank2  1.035965  1.048124 -0.341586
# rank3  0.585521  1.907344 -0.264500
# rank1  2.391899 -1.575462  0.616940

2、分组

Pandas可以使用groupby函数对DataFrame进行拆分，得到分组对象。

df.groupby(self, by=None, axis=0, level=None, as_index=True, sort=True,
            group_keys=True, squeeze=False, observed=False, **kwargs)

by:分组方式，可以是字典、函数、标签、标签列表

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data = [['Alex', 24, 80], ['Bob', 25, 90], ['Bauer', 25, 90], ['Jack', 26, 80]]
    df = pd.DataFrame(data, index=['a', 'b', 'c', 'd'], columns=['Name', 'Age', 'A'])
    print(df)
    group_obj1 = df.groupby('Name')
    print(group_obj1.groups)
    print('===================================')
    # 单层分组迭代
    for key, data in group_obj1:
        print(key)
        print(data)
    group_obj2 = df.groupby(['Name', 'A'])
    # 分组信息查看
    print(group_obj2.groups)
    print('===================================')
    # 多层分组迭代
    for key, data in group_obj2:
        print(key)
        print(data)

# output:
#     Name  Age   A
# a   Alex   24  80
# b    Bob   25  90
# c  Bauer   25  90
# d   Jack   26  80
# {'Alex': Index(['a'], dtype='object'), 'Bauer': Index(['c'], dtype='object'), 'Bob': Index(['b'], dtype='object'), 'Jack': Index(['d'], dtype='object')}
# ===================================
# Alex
#    Name  Age   A
# a  Alex   24  80
# Bauer
#     Name  Age   A
# c  Bauer   25  90
# Bob
#   Name  Age   A
# b  Bob   25  90
# Jack
#    Name  Age   A
# d  Jack   26  80
# {('Alex', 80): Index(['a'], dtype='object'), ('Bauer', 90): Index(['c'], dtype='object'), ('Bob', 90): Index(['b'], dtype='object'), ('Jack', 80): Index(['d'], dtype='object')}
# ===================================
# ('Alex', 80)
#    Name  Age   A
# a  Alex   24  80
# ('Bauer', 90)
#     Name  Age   A
# c  Bauer   25  90
# ('Bob', 90)
#   Name  Age   A
# b  Bob   25  90
# ('Jack', 80)
#    Name  Age   A
# d  Jack   26  80

filter()函数可以用于过滤数据。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data = [['Alex', 24, 80], ['Bob', 25, 92], ['Bauer', 25, 90], ['Jack', 26, 80]]
    df = pd.DataFrame(data, index=['a', 'b', 'c', 'd'], columns=['Name', 'Age', 'A'])
    print(df)
    group_obj1 = df.groupby('Age')
    print(group_obj1.groups)
    # 过滤年龄相同的人
    group = group_obj1.filter(lambda x: len(x) > 1)
    print(group)

# output:
#     Name  Age   A
# a   Alex   24  80
# b    Bob   25  92
# c  Bauer   25  90
# d   Jack   26  80
# {24: Index(['a'], dtype='object'), 25: Index(['b', 'c'], dtype='object'), 26: Index(['d'], dtype='object')}
#     Name  Age   A
# b    Bob   25  92
# c  Bauer   25  90

3、合并

pd.merge(left, right, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=True)
合并两个DataFrame对象。
left ，左DataFrame对象。
right，右DataFrame对象。
on，列(名称)连接，必须在左DataFrame和右DataFrame对象中存在(找到)。
left_on，左侧DataFrame中的列用作键，可以是列名或长度等于DataFrame长度的数组。
right_on，来自右DataFrame的列作为键，可以是列名或长度等于DataFrame长度的数组。
left_index，如果为True，则使用左侧DataFrame中的索引(行标签)作为其连接键。 在具有MultiIndex(分层)的DataFrame的情况下，级别的数量必须与来自右DataFrame的连接键的数量相匹配。
right_index ，与右DataFrame的left_index具有相同的用法。
how，可选值为left, right, outer，inner，默认为inner。
sort，按照字典顺序通过连接键对结果DataFrame进行排序。默认为True，设置为False时，可以大大提高性能。
在一个键上合并两个DataFrame的示例如下：

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [['Alex', 24, 80], ['Bob', 25, 90], ['Bauer', 25, 90]]
    left = pd.DataFrame(data1, columns=['Name', 'Age', 'A'])

    data2 = [['Alex', 87, 78], ['Bob', 67, 87], ['Bauer', 98, 78]]
    right = pd.DataFrame(data2, columns=['Name', 'B', 'C'])

    print(left)
    print('==================================')
    print(right)
    print('==================================')
    df = pd.merge(left, right, on='Name')
    print(df)

# output:
#     Name  Age   A
# 0   Alex   24  80
# 1    Bob   25  90
# 2  Bauer   25  90
# ==================================
#     Name   B   C
# 0   Alex  87  78
# 1    Bob  67  87
# 2  Bauer  98  78
# ==================================
#     Name  Age   A   B   C
# 0   Alex   24  80  87  78
# 1    Bob   25  90  67  87
# 2  Bauer   25  90  98  78

合并多个键上的两个DataFrame的示例如下：

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [[1, 'Alex', 24, 80], [2, 'Bob', 25, 90], [3, 'Bauer', 25, 90]]
    left = pd.DataFrame(data1, columns=['ID', 'Name', 'Age', 'A'])

    data2 = [[1, 'Alex', 87, 78], [4, 'Bob', 67, 87], [3, 'Bauer', 98, 78]]
    right = pd.DataFrame(data2, columns=['ID', 'Name', 'B', 'C'])

    print(left)
    print('==================================')
    print(right)
    print('==================================')
    df = pd.merge(left, right, on=['ID', 'Name'])
    print(df)

# output:
#    ID   Name  Age   A
# 0   1   Alex   24  80
# 1   2    Bob   25  90
# 2   3  Bauer   25  90
# ==================================
#    ID   Name   B   C
# 0   1   Alex  87  78
# 1   4    Bob  67  87
# 2   3  Bauer  98  78
# ==================================
#    ID   Name  Age   A   B   C
# 0   1   Alex   24  80  87  78
# 1   3  Bauer   25  90  98  78

使用“how”参数进行合并，如何合并参数指定如何确定哪些键将被包含在结果表中。如果组合键没有出现在左侧或右侧表中，则连接表中的值将为NA。
left：LEFT OUTER JOIN，使用左侧对象的键。
right：RIGHT OUTER JOIN，使用右侧对象的键。
outer：FULL OUTER JOIN，使用键的联合。
inner：INNER JOIN，使用键的交集。
Left Join示例：

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [[1, 'Alex', 24, 80], [2, 'Bob', 25, 90], [3, 'Bauer', 25, 90]]
    left = pd.DataFrame(data1, columns=['ID', 'Name', 'Age', 'A'])

    data2 = [[1, 'Alex', 87, 78], [4, 'Bob', 67, 87], [3, 'Bauer', 98, 78]]
    right = pd.DataFrame(data2, columns=['ID', 'Name', 'B', 'C'])

    print(left)
    print('==================================')
    print(right)
    print('==================================')
    df = pd.merge(left, right, on='ID', how='left')
    print(df)

# output:
#    ID   Name  Age   A
# 0   1   Alex   24  80
# 1   2    Bob   25  90
# 2   3  Bauer   25  90
# ==================================
#    ID   Name   B   C
# 0   1   Alex  87  78
# 1   4    Bob  67  87
# 2   3  Bauer  98  78
# ==================================
#    ID Name_x  Age   A Name_y     B     C
# 0   1   Alex   24  80   Alex  87.0  78.0
# 1   2    Bob   25  90    NaN   NaN   NaN
# 2   3  Bauer   25  90  Bauer  98.0  78.0

Right Join示例：

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [[1, 'Alex', 24, 80], [2, 'Bob', 25, 90], [3, 'Bauer', 25, 90]]
    left = pd.DataFrame(data1, columns=['ID', 'Name', 'Age', 'A'])

    data2 = [[1, 'Alex', 87, 78], [4, 'Bob', 67, 87], [3, 'Bauer', 98, 78]]
    right = pd.DataFrame(data2, columns=['ID', 'Name', 'B', 'C'])

    print(left)
    print('==================================')
    print(right)
    print('==================================')
    df = pd.merge(left, right, on='ID', how='right')
    print(df)

# output:
#    ID   Name  Age   A
# 0   1   Alex   24  80
# 1   2    Bob   25  90
# 2   3  Bauer   25  90
# ==================================
#    ID   Name   B   C
# 0   1   Alex  87  78
# 1   4    Bob  67  87
# 2   3  Bauer  98  78
# ==================================
#    ID Name_x   Age     A Name_y   B   C
# 0   1   Alex  24.0  80.0   Alex  87  78
# 1   3  Bauer  25.0  90.0  Bauer  98  78
# 2   4    NaN   NaN   NaN    Bob  67  87

Outer Join示例：

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [[1, 'Alex', 24, 80], [2, 'Bob', 25, 90], [3, 'Bauer', 25, 90]]
    left = pd.DataFrame(data1, columns=['ID', 'Name', 'Age', 'A'])

    data2 = [[1, 'Alex', 87, 78], [4, 'Bob', 67, 87], [3, 'Bauer', 98, 78]]
    right = pd.DataFrame(data2, columns=['ID', 'Name', 'B', 'C'])

    print(left)
    print('==================================')
    print(right)
    print('==================================')
    df = pd.merge(left, right, on='ID', how='outer')
    print(df)

# output:
#    ID   Name  Age   A
# 0   1   Alex   24  80
# 1   2    Bob   25  90
# 2   3  Bauer   25  90
# ==================================
#    ID   Name   B   C
# 0   1   Alex  87  78
# 1   4    Bob  67  87
# 2   3  Bauer  98  78
# ==================================
#    ID Name_x   Age     A Name_y     B     C
# 0   1   Alex  24.0  80.0   Alex  87.0  78.0
# 1   2    Bob  25.0  90.0    NaN   NaN   NaN
# 2   3  Bauer  25.0  90.0  Bauer  98.0  78.0
# 3   4    NaN   NaN   NaN    Bob  67.0  87.0

Inner Join示例：

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [[1, 'Alex', 24, 80], [2, 'Bob', 25, 90], [3, 'Bauer', 25, 90]]
    left = pd.DataFrame(data1, columns=['ID', 'Name', 'Age', 'A'])

    data2 = [[1, 'Alex', 87, 78], [4, 'Bob', 67, 87], [3, 'Bauer', 98, 78]]
    right = pd.DataFrame(data2, columns=['ID', 'Name', 'B', 'C'])

    print(left)
    print('==================================')
    print(right)
    print('==================================')
    df = pd.merge(left, right, on='ID', how='inner')
    print(df)

# output:
#    ID   Name  Age   A
# 0   1   Alex   24  80
# 1   2    Bob   25  90
# 2   3  Bauer   25  90
# ==================================
#    ID   Name   B   C
# 0   1   Alex  87  78
# 1   4    Bob  67  87
# 2   3  Bauer  98  78
# ==================================
#    ID Name_x  Age   A Name_y   B   C
# 0   1   Alex   24  80   Alex  87  78
# 1   3  Bauer   25  90  Bauer  98  78

4、级联

concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,
           keys=None, levels=None, names=None, verify_integrity=False,
           sort=None, copy=True)

沿某个轴进行级联操作。
objs，Series、DataFrame或Panel对象的序列或字典。
axis，{0，1，...}，默认为0，axis=0表示按index进行级联，axis=1表示按columns进行级联。
join，{'inner', 'outer'}，默认inner，指示如何处理其它轴上的索引。
ignore_index，布尔值，默认为False。如果指定为True，则不使用连接轴上的索引值。结果轴将被标记为：0，...，n-1。
join_axes ，Index对象的列表。用于其它(n-1)轴的特定索引，而不是执行内部/外部集逻辑。
sort:是否进行排序，True会进行排序，False不进行排序。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [['Alex', 24, 80], ['Bob', 25, 90], ['Bauer', 25, 90]]
    one = pd.DataFrame(data1, columns=['Name', 'Age', 'A'])

    data2 = [['Alex', 87, 78], ['Bob', 67, 87], ['Bauer', 98, 78]]
    two = pd.DataFrame(data2, columns=['Name', 'B', 'C'])

    print(one)
    print('==================================')
    print(two)
    print('==================================')
    df = pd.concat([one, two], axis=1, sort=False)
    print(df)

# output:
#     Name  Age   A
# 0   Alex   24  80
# 1    Bob   25  90
# 2  Bauer   25  90
# ==================================
#     Name   B   C
# 0   Alex  87  78
# 1    Bob  67  87
# 2  Bauer  98  78
# ==================================
#     Name  Age   A   Name   B   C
# 0   Alex   24  80   Alex  87  78
# 1    Bob   25  90    Bob  67  87
# 2  Bauer   25  90  Bauer  98  78

当结果的索引是重复的，如果想要生成的对象必须遵循自己的索引，需要将ignore_index设置为True。
Pandas提供了连接DataFrame的append方法，沿axis=0连接。

df.append(self, other, ignore_index=False,
           verify_integrity=False, sort=None)

向DataFrame对象中添加新的行，如果添加的列名不在DataFrame对象中，将会被当作新的列进行添加。
other：DataFrame、series、dict、list
ignore_index：默认值为False，如果为True则不使用index标签。
verify_integrity ：默认值为False，如果为True当创建相同的index时会抛出ValueError的异常。
sort：boolean，默认是None。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [['Alex', 24, 80], ['Bob', 25, 90], ['Bauer', 25, 90]]
    one = pd.DataFrame(data1, columns=['Name', 'Age', 'A'])

    data2 = [['Alex', 87, 78], ['Bob', 67, 87], ['Bauer', 98, 78]]
    two = pd.DataFrame(data2, columns=['Name', 'B', 'C'])

    print(one)
    print('==================================')
    print(two)
    print('==================================')
    df = one.append(two, sort=False)
    print(df)

# output:
#     Name  Age   A
# 0   Alex   24  80
# 1    Bob   25  90
# 2  Bauer   25  90
# ==================================
#     Name   B   C
# 0   Alex  87  78
# 1    Bob  67  87
# 2  Bauer  98  78
# ==================================
#     Name   Age     A     B     C
# 0   Alex  24.0  80.0   NaN   NaN
# 1    Bob  25.0  90.0   NaN   NaN
# 2  Bauer  25.0  90.0   NaN   NaN
# 0   Alex   NaN   NaN  87.0  78.0
# 1    Bob   NaN   NaN  67.0  87.0
# 2  Bauer   NaN   NaN  98.0  78.0

Pandas提供了连接DataFrame的join方法，沿axis=1连接，用于将两个DataFrame中的不同的列索引合并成为一个DataFrame。

df.join(self, other, on=None, how='left', lsuffix='', rsuffix='',
         sort=False)

join方法提供SQL的Join操作，默认为为左外连接how=left。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data1 = [['Alex', 24, 80], ['Bob', 25, 90], ['Bauer', 25, 90],['Jack', 26, 80]]
    one = pd.DataFrame(data1, index=['a', 'b', 'c', 'd'], columns=['Name', 'Age', 'A'])

    data2 = [[87, 78], [67, 87], [98, 78]]
    two = pd.DataFrame(data2, index=['a', 'b', 'c'], columns=['B', 'C'])

    print(one)
    print('==================================')
    print(two)
    print('==================================')
    df = one.join(two)
    print(df)

# output:
#     Name  Age   A
# a   Alex   24  80
# b    Bob   25  90
# c  Bauer   25  90
# d   Jack   26  80
# ==================================
#     B   C
# a  87  78
# b  67  87
# c  98  78
# ==================================
#     Name  Age   A     B     C
# a   Alex   24  80  87.0  78.0
# b    Bob   25  90  67.0  87.0
# c  Bauer   25  90  98.0  78.0
# d   Jack   26  80   NaN   NaN

5、迭代

迭代DataFrame提供列名。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    print(df)
    for col in df:
        print(col, end=' ')

# output:
#                    A         B         C         D
# 2019-01-01 -0.415754 -1.214340 -0.103952  1.232414
# 2019-01-02 -0.367888  0.257199 -1.615029 -0.335322
# 2019-01-03  0.552697  0.202993 -1.000219 -0.530897
# 2019-01-04  0.503410 -1.610091  1.660362  0.649700
# 2019-01-05  0.575416 -1.962578 -1.681379 -0.425239
# 2019-01-06  1.075917 -0.499081  1.886878 -0.073895
# A B C D

df.iteritems()用于迭代(key，value)对，将每个列标签作为key，value为Series对象。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    dates = pd.date_range('20190101', periods=6)
    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    print(df)
    for key, value in df.iteritems():
        print(key, value)

# output:
#                    A         B         C         D
# 2019-01-01 -0.302021  1.343811 -0.070351 -0.409479
# 2019-01-02 -0.365564  0.743572 -0.475075  1.026054
# 2019-01-03  0.025748  1.395340 -0.987686  0.141003
# 2019-01-04 -0.291348 -1.173600 -2.286905  0.528416
# 2019-01-05 -1.844523 -0.052567  0.575980  0.260001
# 2019-01-06  0.271046 -0.583334 -0.596251  0.772095
# A 2019-01-01   -0.302021
# 2019-01-02   -0.365564
# 2019-01-03    0.025748
# 2019-01-04   -0.291348
# 2019-01-05   -1.844523
# 2019-01-06    0.271046
# Freq: D, Name: A, dtype: float64
# B 2019-01-01    1.343811
# 2019-01-02    0.743572
# 2019-01-03    1.395340
# 2019-01-04   -1.173600
# 2019-01-05   -0.052567
# 2019-01-06   -0.583334
# Freq: D, Name: B, dtype: float64
# C 2019-01-01   -0.070351
# 2019-01-02   -0.475075
# 2019-01-03   -0.987686
# 2019-01-04   -2.286905
# 2019-01-05    0.575980
# 2019-01-06   -0.596251
# Freq: D, Name: C, dtype: float64
# D 2019-01-01   -0.409479
# 2019-01-02    1.026054
# 2019-01-03    0.141003
# 2019-01-04    0.528416
# 2019-01-05    0.260001
# 2019-01-06    0.772095
# Freq: D, Name: D, dtype: float64

df.iterrows()用于返回迭代器，产生每个index以及包含每行数据的Series。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
    print(df)
    for index, value in df.iterrows():
        print(index, value)

# output:
#           A         B         C         D
# 0 -1.097851  0.785749 -1.727198 -1.120925
# 1 -1.420429  0.094384 -1.566202  0.237084
# 2 -0.761957  0.552395  0.680884 -0.290955
# 3  0.357713 -0.323331  1.438013 -1.334616
# 4  0.015467 -2.431556 -0.717285 -0.094409
# 5 -1.198224 -1.370170  0.201725  0.258093
# 0 A   -1.097851
# B    0.785749
# C   -1.727198
# D   -1.120925
# Name: 0, dtype: float64
# 1 A   -1.420429
# B    0.094384
# C   -1.566202
# D    0.237084
# Name: 1, dtype: float64
# 2 A   -0.761957
# B    0.552395
# C    0.680884
# D   -0.290955
# Name: 2, dtype: float64
# 3 A    0.357713
# B   -0.323331
# C    1.438013
# D   -1.334616
# Name: 3, dtype: float64
# 4 A    0.015467
# B   -2.431556
# C   -0.717285
# D   -0.094409
# Name: 4, dtype: float64
# 5 A   -1.198224
# B   -1.370170
# C    0.201725
# D    0.258093
# Name: 5, dtype: float64

df.itertuples()方法将为DataFrame中的每一行返回一个产生一个命名元组的迭代器。元组的第一个元素是行的index，而剩余的值是行值。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
    print(df)
    for row in df.itertuples():
        print(row)

# output:
#           A         B         C         D
# 0  0.681324  1.047734 -1.909570 -0.845900
# 1 -0.879077 -0.897085 -0.795461 -0.634519
# 2  0.484502 -0.061608  0.605827 -0.321721
# 3 -0.051974  1.533112 -1.011544 -0.922280
# 4 -0.634157 -0.173692  1.228584 -1.229581
# 5  0.236769 -0.933609  0.111948  1.048215
# Pandas(Index=0, A=0.6813238552921729, B=1.0477343302788706, C=-1.909570436815022, D=-0.8459001766064564)
# Pandas(Index=1, A=-0.8790771200969485, B=-0.8970849190216943, C=-0.7954606477323869, D=-0.6345188867416923)
# Pandas(Index=2, A=0.48450157948338324, B=-0.061608014575315506, C=0.6058267522125123, D=-0.32172144100965605)
# Pandas(Index=3, A=-0.05197447447575398, B=1.5331115391025778, C=-1.0115444345763995, D=-0.9222798204619236)
# Pandas(Index=4, A=-0.6341570074338677, B=-0.173692444412635, C=1.2285839004083785, D=-1.2295807166909738)
# Pandas(Index=5, A=0.23676890089548117, B=-0.9336090868233837, C=0.11194794444517034, D=1.0482154173833818)

迭代用于读取，迭代器返回原始对象(视图)的副本，因此迭代时更改将不会反映在原始对象上。

6、SQL化操作

在SQL中，SELECT使用逗号分隔的列列表(或选择所有列)来完成。
SELECT ID, Name FROM tablename LIMIT 5;
在Pandas中，列选择通过传递列名到DataFrame。
df[['ID', 'Name']].head(5)
SELECT操作示例：

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data = [[1, 'Alex', 24, 80], [2, 'Bob', 25, 90], [3, 'Bauer', 25, 90]]
    df = pd.DataFrame(data, columns=['ID', 'Name', 'Age', 'A'])
    print(df)
    print(df[['ID', 'Name']].head(5))

# output:
#    ID   Name  Age   A
# 0   1   Alex   24  80
# 1   2    Bob   25  90
# 2   3  Bauer   25  90
#    ID   Name
# 0   1   Alex
# 1   2    Bob
# 2   3  Bauer

在SQL中，使用WHERE进行条件过滤。
SELECT * FROM tablename WHERE Name = 'Bauer' LIMIT 5;
在Pandas中，通常使用布尔索引进行过滤。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    data = [[1, 'Alex', 24, 80], [2, 'Bob', 25, 90], [3, 'Bauer', 25, 90]]
    df = pd.DataFrame(data, columns=['ID', 'Name', 'Age', 'A'])
    print(df)
    print('===========================')
    print(df[df['Name'] == 'Bauer'].head(5))

# output:
#    ID   Name  Age   A
# 0   1   Alex   24  80
# 1   2    Bob   25  90
# 2   3  Bauer   25  90
# ===========================
#    ID   Name  Age   A
# 2   3  Bauer   25  90

四、数据分析

1、描述性统计

（1）sum
返回所请求轴的值的总和。 默认情况下，轴为索引(axis=0)。

import pandas as pd

if __name__ == "__main__":
    data = [['Alex', 25, 80], ['Bob', 26, 90], ['Bauer', 24, 87]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df.sum())
    print(df.sum(1))

# output:
#     Name  Age  Score
# 0   Alex   25     80
# 1    Bob   26     90
# 2  Bauer   24     87
# Name     AlexBobBauer
# Age                75
# Score             257
# dtype: object
# 0    105
# 1    116
# 2    111
# dtype: int64

（2）mean
返回平均值。

import pandas as pd

if __name__ == "__main__":
    data = [['Alex', 25, 80], ['Bob', 26, 90], ['Bauer', 24, 87]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df.mean())

# output:
#     Name  Age  Score
# 0   Alex   25     80
# 1    Bob   26     90
# 2  Bauer   24     87
# Age      25.000000
# Score    85.666667
# dtype: float64

（3）std
返回数字列的Bressel标准偏差。

import pandas as pd

if __name__ == "__main__":
    data = [['Alex', 25, 80], ['Bob', 26, 90], ['Bauer', 24, 87]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df.std())

# output:
#     Name  Age  Score
# 0   Alex   25     80
# 1    Bob   26     90
# 2  Bauer   24     87
# Age      1.000000
# Score    5.131601
# dtype: float64

（4）median
求所有值的中位数。

import pandas as pd

if __name__ == "__main__":
    data = [['Alex', 25, 80], ['Bob', 26, 90], ['Bauer', 24, 87]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df.median())

# output:
#     Name  Age  Score
# 0   Alex   25     80
# 1    Bob   26     90
# 2  Bauer   24     87
# Age      25.0
# Score    87.0
# dtype: float64

（5）min
求所有值中的最小值。

import pandas as pd

if __name__ == "__main__":
    data = [['Alex', 25, 80], ['Bob', 26, 90], ['Bauer', 24, 87]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df.min())

# output:
#     Name  Age  Score
# 0   Alex   25     80
# 1    Bob   26     90
# 2  Bauer   24     87
# Name     Alex
# Age        24
# Score      80
# dtype: object

（6）max
求所有值中的最大值。

import pandas as pd

if __name__ == "__main__":
    data = [['Alex', 25, 80], ['Bob', 26, 90], ['Bauer', 24, 87]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df.max())

# output:
#     Name  Age  Score
# 0   Alex   25     80
# 1    Bob   26     90
# 2  Bauer   24     87
# Name     Bob
# Age       26
# Score     90
# dtype: object

（7）describe
汇总有关DataFrame列的统计信息的摘要。
def describe(self, percentiles=None, include=None, exclude=None)
include用于传递关于什么列需要考虑用于总结的必要信息的参数。获取值列表，默认情况下是number 。
object - 汇总字符串列
number - 汇总数字列
all - 将所有列汇总在一起(不应将其作为列表值传递)

import pandas as pd

if __name__ == "__main__":
    data = [['Alex', 25, 80], ['Bob', 26, 90], ['Bauer', 24, 87]]
    df = pd.DataFrame(data, columns=['Name', 'Age', 'Score'])
    print(df)
    print(df.describe(include="all"))

# output:
# Name  Age  Score
# 0   Alex   25     80
# 1    Bob   26     90
# 2  Bauer   24     87
#         Name   Age      Score
# count      3   3.0   3.000000
# unique     3   NaN        NaN
# top     Alex   NaN        NaN
# freq       1   NaN        NaN
# mean     NaN  25.0  85.666667
# std      NaN   1.0   5.131601
# min      NaN  24.0  80.000000
# 25%      NaN  24.5  83.500000
# 50%      NaN  25.0  87.000000
# 75%      NaN  25.5  88.500000
# max      NaN  26.0  90.000000

abs：求所有值的绝对值
prod：求所有值的乘积
cumsum：累计总和
cumprod：累计乘积

2、百分比变化

Series，DatFrames和Panel都有pct_change()函数，用于将每个元素与其前一个元素进行比较，并计算变化百分比。默认情况下，pct_change()对列进行操作; 如果想应用到行上，那么可使用axis = 1参数。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(4, 3), index=['rank2', 'rank1', 'rank4', 'rank3'], columns=['col3', 'col2', 'col1'])
    print(df)
    print(df.pct_change())

# output:
#            col3      col2      col1
# rank2  0.988739  2.062798  1.400892
# rank1  0.394663 -0.988307  1.583098
# rank4 -0.768109 -0.163727 -1.801323
# rank3  0.999816 -1.224068  1.470020
#            col3      col2      col1
# rank2       NaN       NaN       NaN
# rank1 -0.600842 -1.479110  0.130064
# rank4 -2.946241 -0.834336 -2.137846
# rank3 -2.301659  6.476294 -1.816078

3、协方差

协方差适用于Series数据，Series对象有一个方法cov用来计算Series对象之间的协方差，NA将被自动排除。当应用于DataFrame对象时，协方差方法计算所有列之间的协方差(cov)值。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(3, 5), columns=['a', 'b', 'c', 'd', 'e'])
    print(df)
    print(df['a'].cov(df['b']))
    print(df.cov())

# output:
#           a         b         c         d         e
# 0  1.168443 -0.343905  2.254448  0.269765 -0.928009
# 1  0.542551 -1.303205 -1.767313 -0.349884 -0.352578
# 2 -2.028410 -1.176339  0.156047  1.426468 -1.338805
# 0.48923631972868176
#           a         b         c         d         e
# a  2.870241  0.489236  0.713430 -1.312818  0.581441
# b  0.489236  0.271550  0.974811 -0.023849 -0.055862
# c  0.713430  0.974811  4.046193  0.580236 -0.558184
# d -1.312818 -0.023849  0.580236  0.812892 -0.430603
# e  0.581441 -0.055862 -0.558184 -0.430603  0.245420

4、相关性

相关性显示了任何两个数值(Series)之间的线性关系。有多种计算相关性的方法，如pearson(默认)，spearman和kendall。如果DataFrame中存在任何非数字列，则会自动排除。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    df = pd.DataFrame(np.random.randn(3, 5), columns=['a', 'b', 'c', 'd', 'e'])
    print(df)
    print(df['a'].corr(df['b']))
    print(df.corr())

# output:
#           a         b         c         d         e
# 0 -2.110756  0.693665  0.405701 -0.628349 -1.062029
# 1 -1.331364  1.283434  1.619166 -0.025866  1.742287
# 2 -1.159944  0.435840 -0.251710 -0.347102 -0.026825
# 0.052396578025987336
#           a         b         c         d         e
# a  1.000000  0.052397 -0.000006  0.743940  0.664845
# b  0.052397  1.000000  0.998626  0.706309  0.780790
# c -0.000006  0.998626  1.000000  0.668242  0.746977
# d  0.743940  0.706309  0.668242  1.000000  0.993772
# e  0.664845  0.780790  0.746977  0.993772  1.000000

5、数据排名

数据排名为元素数组中的每个元素生成排名。在关系的情况下，分配平均等级。

# -*- coding=utf-8 -*-
import pandas as pd
import numpy as np

if __name__ == "__main__":
    s = pd.Series(np.random.randn(5), index=list('abcde'))
    print(s)
    s['a'] = s['c']
    print(s.rank())

# output:
# a    1.597684
# a    1.597684
# b    1.107413
# c   -0.298296
# d   -0.281076
# e   -0.667954
# dtype: float64
# a    2.5
# b    5.0
# c    2.5
# d    4.0
# e    1.0
# dtype: float64

rank使用一个默认为True的升序参数; False时，数据被反向排序，较大的值被分配较小的排序。 


分享文章：Python3快速入门（十五）——Pandas数据处理
分享路径：http://scyingshan.cn/article/josged.html

其他资讯

快速导航
友情链接
网站建设
联系方式
  • 企业：青羊区大橙子信息咨询工作室
  • 地址：成都市青羊区太升南路288号
  • 电话：18980820575
  • 网址：www.scyingshan.cn

公司名称：青羊区大橙子信息咨询工作室 联系电话：18980820575

网站备案号：蜀ICP备2022028542号-24

营山大橙子建站 营山网站建设 营山网站设计 营山网站制作 成都做网站