导入错误：无法导入名称sqlContext

Question

我使用pyspark读一些CSV数据引发数据帧

，当我尝试导入pyspark模块如下： 从pyspark.sql进口sqlContext

为什么我得到以下错误？如何解决它？ 导入错误：无法导入名称sqlContext

我使用Python 2.7和2.0.1星火

非常感谢！

Answer 1

这可能是因为你没有正确设置python pat。在配置我的Python环境时，我发现以下功能很有用。

def configure_spark(spark_home=None, pyspark_python=None, conf_dir=None): 
    """Configures the Python path for importing pyspark 

    Sets the SPARK_HOME and PYSPARK_PYTHON environment variables and modifies 
    the Python PATH so the pyspark package can be imported. 

    Args: 
     spark_home (str): Path of SPARK_HOME. Defaults to SPARK_HOME module 
      variable. 
     pyspark_python (str): Path to Python binary to use in PySpark. Defaults 
      to the currently executing Python binary. 
     conf_dir (str): Path to configuration directory 
    """ 

    # Set the configuration directory with some basic sanity checks: 
    if conf_dir: 
     if not os.path.isdir(conf_dir): 
      raise OSError("Spark config directory not found: %s" % conf_dir) 

     expected_conf = {'spark-env.sh', 'spark-defaults.conf'} 
     found_conf = expected_conf - set(os.listdir(conf_dir)) 
     if found_conf: 
      warnings.warn("Some configuration files were not found: %s" % found_conf) 

     os.environ['SPARK_CONF_DIR'] = conf_dir 

    spark_home = spark_home or SPARK_HOME 
    os.environ['SPARK_HOME'] = spark_home 

    if not os.path.isdir(spark_home): 
     raise OSError("Specified SPARK_HOME is not a valid directory: %s" % spark_home) 

    # Add the PySpark directories to the Python path: 
    libs = glob(os.path.join(spark_home, 'python', 'lib', '*.zip')) 
    if len(libs) < 2: 
     raise OSError("Pyspark libraries not found in %s" % spark_home) 
    for lib in libs: 
     sys.path.insert(1, lib) 

    # If PYSPARK_PYTHON isn't specified, use currently running Python binary: 
    pyspark_python = pyspark_python or sys.executable 
    os.environ['PYSPARK_PYTHON'] = pyspark_python